diff --git a/README.md b/README.md index e11b321..780afc2 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,8 @@ The Python Shapefile Library (PyShp) reads and writes ESRI Shapefiles in pure Py - **Author**: [Joel Lawhead](https://github.com/GeospatialPython) - **Maintainers**: [James Parrott](https://github.com/JamesParrott) & [Karim Bahgat](https://github.com/karimbahgat) -- **Version**: 3.1.6 -- **Date**: 25th July 2026 +- **Version**: 3.1.7.dev +- **Date**: 6th Nov 2026 - **License**: [MIT](https://github.com/GeospatialPython/pyshp/blob/master/LICENSE.TXT) ## Contents @@ -91,8 +91,8 @@ reading and writing data, and allows you to focus on the challenging and FUN part of your geospatial project. -# Version Changes - +# Recent version Changes +See also the Older Version Changes section below ## 3.1.6 ### Feature - Encodings can now be read from .cpg files (and optionally written to them). @@ -101,1788 +101,1791 @@ part of your geospatial project. ### Bug fix - Fixed another bug causing dates before the year 1000 to be encoded as less than 8 chars under "%Y%m%d (found by [Thomas Beierlein](https://github.com/GeospatialPython/pyshp/issues/435)) -## 3.1.4 -### Bug fix - - Fix bug causing dates supplied as length 8 strings of digits to be encoded by the custom encoding, not ascii. -### Testing - - Test other codecs (ascii and UTF-8, UTF-16 & UTF-32 so far). - - Test all available codecs in CI (92 of them). -## 3.1.3 - - Restore faster text writing paths for single-byte Ascii encodings, and Utf-8. +# The Basics -## 3.1.2 - - Raise error in strict mode when creating field with a name, or writing strings, that ends with whole code points - that whose encoding is pad bytes. +Before doing anything you must import the library. -## 3.1.1 -### Unicode support made even more robust and yet another encoding bug fixed! - - When reading, only use minimum number of pad bytes to decode text successfully (fixes issue 423). - - When writing, warn (or raise in strict mode) if the text's encoding ends in pad bytes. -## 3.1.0 -### Unicode support made more robust and encoding bugs fixed - - Truncation of field names and text fields now respects unicode code point boundaries (fixes issues - - 416 and 148). - - Warnings raised if truncation, or replacing b" " with b"_" would corrupt encoded field names, - - both if they would either be undecodable, or would silently decode to incorrect data (warns users - if issue 421 applies). - - Correctly truncated field names are now stored in field instances, as would actually be seen in the file. - - New strict mode. Writer(strict=True) raises errors or refuse to create fields and text records with data that - - would be truncated or cannot be correctly decoded back again by PyShp, exactly as given by the user. - - In strict mode, ascii spaces in encoded names are no longer replaced by ascii underscores at all - (work around to avoid corrupting unicode field names - provides opt-in fix for issue 421). - - BREAKING. When reading .dbf files, Trailing ascii spaces in text fields before a null terminator char (in the - - decoded string) is now removed (i.e. instead of .strip().rstrip('\x00') we now do: .rstrip("\x00").rstrip(" ")). - - BREAKING. Enclosing Whitespace other than trailing ascii spaces (0x20) after null chars in text fields is now - - preserved, when reading .dbf files (fixes issue 418 - James feels this was a bug. Let him know if you think otherwise). - - BREAKING Trailing null chars other than null terminators & null padding bytes, followed by whitespace other than - ascii spaces, are now preserved. - - Writing dbf records is now atomic. + >>> import shapefile -### ShpWriter.shape API Tweak (small breaking change). - - Make ShpWriter.shape return shape length in bytes (the - same as for offset) not in 16 bit words. -### Testing - - Include NullShapes in shp round trip test. - - En/decoding of Dbf files and Fields round trips correctly. +The examples below will use a shapefile created from the U.S. Census Bureau +Blockgroups data set near San Francisco, CA and available in the git +repository of the PyShp GitHub site. +## Reading Shapefiles -## 3.0.13 -### Bug fix - - Fix bug when reading empty shp files. -### Testing - - Add round trip tests for Multipatch and shp files (both passed). +### The Reader Class -## 3.0.12 -### Data consistency - - Add Shape.points_2D and Shape.points_3D properties - lists of guaranteed length tuples (2 and 3 respectively). +#### Reading Shapefiles from Local Files -### Testing - - Round trip property-based tests for Polylines and Polygons (both pass). +To read a shapefile create a new "Reader" object and pass it the name of an +existing shapefile. The shapefile format is actually a collection of three +files. You specify the base filename of the shapefile or the complete filename +of any of the shapefile component files. -## 3.0.11 -### Edge case handling - - Raise ShapefileException i) when creating Non-null Shapes without (or with empty) points - and ii) when creating Null Shapes with non-empty points. - - Ensure Shape.z and Shape.partTypes are _Arrays. - - Make Shape stricter about its args, e.g. only points or lines, only one point for Points. -### Bug fixes - - Multipoints with only a single point, now have their bbox calculated. + >>> sf = shapefile.Reader("tests/shapefiles/blockgroups") -### Testing - - Round trip property-based tests for Multipoints (passes). +OR -## 3.0.10 -### Bug fix - - Convert directly supplied m values to None if they are strictly below ISDATA_LOWER_BOUND (-1e38). -### Testing - - Move tests into ./tests. - - Remove doctest runner from user land. - - Add round trip property tests for Point and PointM using hypothesis. -## 3.0.9 -### Testing - - Try to make tests not rely on downloads from Github repo URLs, to avoid 404s & 426s due to rate limits. + >>> sf = shapefile.Reader("tests/shapefiles/blockgroups.shp") -### Testability / separation of concerns: - - Separate shx and shp only methods into new ShpWriter, ShxWriter, ShpReader, and ShxReader classes -(instances of each pair is owned by the regular Shapefile Writer and Reader classes respectively). +OR -### Bug fixes - - Remove ambiguity in API (BREAKING CHANGE). Reader and Writer now both raise TypeError - when passed a supported Shapefile target - as the positional-only, non-keyword arg, but also passed kwargs for shp, dbf or shx (instead - of silently dropping or overwriting those kwargs, with the user being none the wiser). - - Bug fix (BREAKING CHANGE). Reader and Writer now raise TypeError instead of continuing silently - as if given no args (e.g. in anticipation of a subsequent call to Reader.load), when passed an unsupported - truthy Python object (not a str or a Path) as its first arg (the positional-only, non-keyword arg). + >>> sf = shapefile.Reader("tests/shapefiles/blockgroups.dbf") - - BREAKING CHANGE. Correct NODATA lower threshold to -1e38 (as per [spec](https://www.esri.com/content/dam/esrisites/sitecore-archive/Files/Pdfs/library/whitepapers/pdfs/shapefile.pdf) 2, Numeric Types). This is 10% of - PyShp's previous threshold -10e38, so it is possible large negative values x (-1e39 < x < -1e38 will now be NODATA. +OR any of the other 5+ formats which are potentially part of a shapefile. The +library does not care about file extensions. You can also specify that you only +want to read some of the file extensions through the use of keyword arguments: -### Security - - Add python_version >= 3.10 to pre-commit dev & lint groups' 'dependency' (due to CVE in filelock 3.19). + >>> sf = shapefile.Reader(dbf="tests/shapefiles/blockgroups.dbf") -## 3.0.8 +#### Reading Shapefiles from Zip Files -### Testability / separation of concerns. - - Separate dbf only writing methods into a new dbfWriter class (an instance of which is owned by the regular Shapefile Writer class). +If your shapefile is wrapped inside a zip file, the library is able to handle that too, meaning you don't have to worry about unzipping the contents: -## 3.0.7 -### Testability / separation of concerns. - - Separate dbf only reading methods into a new dbfReader class (an instance of which is owned by the regular Shapefile Reader class). + >>> sf = shapefile.Reader("tests/shapefiles/blockgroups.zip") +If the zip file contains multiple shapefiles, just specify which shapefile to read by additionally specifying the relative path after the ".zip" part: -## 3.0.6 -### URL Downloading - - Unify tempfile creation and shapefile download logic. - - Check "Content-Type" header and sniff initial bytes - of response in order to possibly reject html responses, before parsing as a shapefile, to give a more useful error to users. - - Special case shapefiles hosted in Github repos to suggest appending the query string `?raw=true`. + >>> sf = shapefile.Reader("tests/shapefiles/blockgroups_multishapefile.zip/blockgroups2.shp") -### Testing: - - Add shapefile from Open Natual Hazard Modelling ([Paula Spannring](https://github.com/PaulaSp3) - and [Felix Oesterle, Austrian Research Centre for Forests](https://orcid.org/0000-0002-7772-6884)). +#### Reading Shapefiles from URLs -## 3.0.5 +Finally, you can use all of the above methods to read shapefiles directly from the internet, by giving a url instead of a local path, e.g.: -### Project structure: - - Use dependency groups for dev and test instead of optional-dependencies in pyproject.toml ([Mike Toews](https://github.com/mwtoews)) + >>> # from a zipped shapefile on website + >>> sf = shapefile.Reader("https://github.com/JamesParrott/PyShp_test_shapefile/raw/refs/heads/main/gadm41_ATA_shp.zip") -## 3.0.4 + >>> # from a shapefile collection of files in a github repository + >>> sf = shapefile.Reader("https://pubs.usgs.gov/of/2000/of00-006/gisdata/coverage/airports.shp") -### Type checking - - Fix pyshp-stubs, and add test to it. Remove all actual stubs from it, and replace with empty "shapefile-stubs/py.typed". Closes Issue 382 (https://github.com/GeospatialPython/pyshp/issues/382) and discussion 371 (https://github.com/GeospatialPython/pyshp/discussions/371). Bug originally found by Thomas Beierlein (https://github.com/GeospatialPython/pyshp/issues/367) +This will automatically download the file(s) to a temporary location before reading, saving you a lot of time and repetitive boilerplate code when you just want quick access to some external data. +#### Reading Shapefiles from File-Like Objects -## 3.0.3 +You can also load shapefiles from any Python file-like object using keyword +arguments to specify any of the three files. This feature is very powerful and +allows you to custom load shapefiles from arbitrary storage formats, such as a protected url or zip file, a serialized object, or in some cases a database. -### Type checking -- Add optional dependency, "stubs", containing the package "pyshp-stubs" generated with stubgen (to avoid -including py.typed, and no longer being a single file project). -### Bug fix -- Prevented UnboundLocalError when reading non-single point M and Z type Shapefiles (@ekawas-vrify). + >>> myshp = open("tests/shapefiles/blockgroups.shp", "rb") + >>> mydbf = open("tests/shapefiles/blockgroups.dbf", "rb") + >>> r = shapefile.Reader(shp=myshp, dbf=mydbf) -### Testing. -- Test PyShp on the Python 3.14 official release (officially released this week, no longer in beta). +Notice in the examples above the shx file is never used. The shx file is a +very simple fixed-record index for the variable-length records in the shp +file. This file is optional for reading. If it's available PyShp will use the +shx file to access shape records a little faster but will do just fine without +it. -## 3.0.2 +#### Reading Shapefiles Using the Context Manager -### .post1 -- Update version at the top of this Readme file (to make what PyPi users see consistent with the changelog and __version__). +The "Reader" class can be used as a context manager, to ensure open file +objects are properly closed when done reading the data: -### .post0 -- Re-release to trigger the deploy job to publish PyShp to Pypi + >>> with shapefile.Reader("tests/shapefiles/blockgroups.shp") as shp: + ... print(shp) + shapefile Reader + 663 shapes (type 'POLYGON') + 663 records (44 fields) -### Bug fix -- Deleted py.typed (probably temporarily). Including the py.typed marker file with a single file package caused type checkers to type check all other libraries installed in the same directory (whether they're typed or not, enforcing type checking for all the adjacent dirs libraries, regardless of the user's intentions for the type checker) (pointed out by @dl1jbe - thanks Thomas). Discussions will be started about a longer term fix, possibly e.g. possibly refactoring to a package, and restoring py.typed. +#### Reading Shapefile Meta-Data -### Code quality: -- Ruff check's UP rule added (mimicks PyUpgrade) (@mwtoews). +Shapefiles have a number of attributes for inspecting the file contents. +A shapefile is a container for a specific type of geometry, and this can be checked using the +shapeType attribute. -## 3.0.1 + >>> sf = shapefile.Reader("tests/shapefiles/blockgroups.dbf") + >>> sf.shapeType + 5 -### Improvements -- Reader(shp=, dbf=, shx=) now support pathlib.Paths, and any pathlike object (@mwtoews). -### Bug fixes -- PyShp 3 no longer modifies the global doctest module (@JamesParrott). -### Code quality -- isort replaced by Ruff check's I rule (@mwtoews). -- mypy --strict used in CI (@JamesParrott). -- LICENSE.TXT re-encoded in UTF-8 (@musicinmybrain). +Shape types are represented by numbers between 0 and 31 as defined by the +shapefile specification and listed below. It is important to note that the numbering system has +several reserved numbers that have not been used yet, therefore the numbers of +the existing shape types are not sequential: -## 3.0.0 +- NULL = 0 +- POINT = 1 +- POLYLINE = 3 +- POLYGON = 5 +- MULTIPOINT = 8 +- POINTZ = 11 +- POLYLINEZ = 13 +- POLYGONZ = 15 +- MULTIPOINTZ = 18 +- POINTM = 21 +- POLYLINEM = 23 +- POLYGONM = 25 +- MULTIPOINTM = 28 +- MULTIPATCH = 31 -### Breaking Changes: -- Python 2 and Python 3.8 support dropped. -- Field info tuple is now a namedtuple (Field) instead of a list. -- Field type codes are now FieldType 'enum' members. -- bbox, mbox and zbox attributes are all new Namedtuples. -- Writer does not mutate Shapes. -- New custom subclasses for each shape type: Null, Multipatch, Point, Polyline, - Multipoint, and Polygon, plus the latter 4's M and Z variants (Reader and - Writer are still compatible with their base class, Shape, as before). -- Shape sub classes are creatable from, and serializable to bytes streams, - as per the shapefile spec. +Based on this we can see that our blockgroups shapefile contains +Polygon type shapes. The shape types are also defined as constants in +the shapefile module, so that we can compare types more intuitively: -### Improvements: -- Speeded up writing shapefiles by up to another ~27% (on top of the recent ~39% improvement in 2.4.1). -### Code quality -- Statically typed, and checked with Mypy -- Checked with Ruff. -- f-strings -- Remove Python 2 specific functions. -- Run doctests against wheels. -- Testing of wheels before publishing them -- pyproject.toml src layout -- Slow test marked. + >>> sf.shapeType == shapefile.POLYGON + True -## 2.4.1 +For convenience, you can also get the name of the shape type as a string: -### Improvements: -- Speed up writing shapefiles by up to ~39%. Combined for loops of calls to f.write(pack(...)), into single calls. -### Breaking Change. Support for Python 2 and Pythons <= 3.8 to be dropped. -- PyShp 2.4.1 is the latest (and likely last) version of PyShp to support Python 2.7 and Pythons <= 3.8. -These CPython versions have reached [end of life](https://devguide.python.org/versions/#versions). -- Future development will focus on PyShp v3.0.0 onwards (currently intended to supporting Pythons >= 3.9). -- This will not break any projects, as pip and other package managers should not install PyShp 3.0.0 -(after its release) in unsupported Pythons. But we no longer promise such projects will get PyShp's latest -bug fixes and features. -- If this negatively impacts your project, all feedback about this decision is welcome -on our [the discussion page](https://github.com/GeospatialPython/pyshp/discussions/290). + >>> sf.shapeTypeName == 'POLYGON' + True -## 2.4.0 - -### New Features: -- Reader.iterRecords now allows start and stop to be specified, to lookup smaller ranges of records. -- Equality comparisons between Records now also require the fields to be the same (and in the same order). +Other pieces of meta-data that we can check include the number of features +and the bounding box area the shapefile covers: -### Development: -- Code quality tools (Ruff format) run on PyShp -- Network, non-network, or all doctests selectable via command line args -- Network tests made runnable on localhost. -## 2.3.1 + >>> len(sf) + 663 + >>> sf.bbox + BBox(xmin=-122.515048, ymin=37.652916, xmax=-122.327622, ymax=37.863433) -### Bug fixes: +Finally, if you would prefer to work with the entire shapefile in a different +format, you can convert all of it to a GeoJSON dictionary, although you may lose +some information in the process, such as z- and m-values: -- Fix recently introduced issue where Reader/Writer closes file-like objects provided by user (#244) -## 2.3.0 + >>> sf.__geo_interface__['type'] + 'FeatureCollection' -### New Features: +### Reading Geometry -- Added support for pathlib and path-like shapefile filepaths (@mwtoews). -- Allow reading individual file extensions via filepaths. +A shapefile's geometry is the collection of points or shapes made from +vertices and implied arcs representing physical locations. All types of +shapefiles just store points. The metadata about the points determine how they +are handled by software. -### Improvements: +You can get a list of the shapefile's geometry by calling the shapes() +method. -- Simplified setup and deployment (@mwtoews) -- Faster shape access when missing shx file -- Switch to named logger (see #240) -### Bug fixes: + >>> shapes = sf.shapes() -- More robust handling of corrupt shapefiles (fixes #235) -- Fix errors when writing to individual file-handles (fixes #237) -- Revert previous decision to enforce geojson output ring orientation (detailed explanation at https://github.com/SciTools/cartopy/issues/2012) -- Fix test issues in environments without network access (@sebastic, @musicinmybrain). +The shapes method returns a list of Shape objects describing the geometry of +each shape record. -## 2.2.0 -### New Features: + >>> len(shapes) + 663 -- Read shapefiles directly from zipfiles. -- Read shapefiles directly from urls. -- Allow fast extraction of only a subset of dbf fields through a `fields` arg. -- Allow fast filtering which shapes to read from the file through a `bbox` arg. +To read a single shape by calling its index use the shape() method. The index +is the shape's count from 0. So to read the 8th shape record you would use its +index which is 7. -### Improvements: -- More examples and restructuring of README. -- More informative Shape to geojson warnings (see #219). -- Add shapefile.VERBOSE flag to control warnings verbosity (default True). -- Shape object information when calling repr(). -- Faster ring orientation checks, enforce geojson output ring orientation. + >>> s = sf.shape(7) + >>> s + Polygon #7 -### Bug fixes: -- Remove null-padding at end of some record character fields. -- Fix dbf writing error when the number of record list or dict entries didn't match the number of fields. -- Handle rare garbage collection issue after deepcopy (https://github.com/mattijn/topojson/issues/120) -- Fix bug where records and shapes would be assigned incorrect record number (@karanrn) -- Fix typos in docs (@timgates) + >>> # Read the bbox of the 8th shape to verify + >>> s.bbox + BBox(xmin=-122.449637, ymin=37.80149, xmax=-122.442109, ymax=37.807958) + >>> # Round coordinates to 3 decimal places + >>> [f'{coord:.3f}' for coord in s.bbox] + ['-122.450', '37.801', '-122.442', '37.808'] -## 2.1.3 +Each shape record (except Points) contains the following attributes. Records of +shapeType Point do not have a bounding box 'bbox'. -### Bug fixes: + >>> for name in dir(shapes[3]): + ... if not name.startswith('_'): + ... name + 'bbox' + 'from_byte_stream' + 'oid' + 'parts' + 'points' + 'points_2D' + 'points_3D' + 'shapeType' + 'shapeTypeName' + 'write_to_byte_stream' -- Fix recent bug in geojson hole-in-polygon checking (see #205) -- Misc fixes to allow geo interface dump to json (eg dates as strings) -- Handle additional dbf date null values, and return faulty dates as unicode (see #187) -- Add writer target typecheck -- Fix bugs to allow reading shp/shx/dbf separately -- Allow delayed shapefile loading by passing no args -- Fix error with writing empty z/m shapefile (@mcuprjak) -- Fix signed_area() so ignores z/m coords -- Enforce writing the 11th field name character as null-terminator (only first 10 are used) -- Minor README fixes -- Added more tests + * `oid`: The shape's index position in the original shapefile. -## 2.1.2 -### Bug fixes: + >>> shapes[3].oid + 3 -- Fix issue where warnings.simplefilter('always') changes global warning behavior [see #203] + * `shapeType`: an integer representing the type of shape as defined by the + shapefile specification. -## 2.1.1 -### Improvements: + >>> shapes[3].shapeType + 5 -- Handle shapes with no coords and represent as geojson with no coords (GeoJSON null-equivalent) -- Expand testing to Python 3.6, 3.7, 3.8 and PyPy; drop 3.3 and 3.4 [@mwtoews] -- Added pytest testing [@jmoujaes] + * `shapeTypeName`: a string representation of the type of shape as defined by shapeType. Read-only. -### Bug fixes: -- Fix incorrect geo interface handling of multipolygons with complex exterior-hole relations [see #202] -- Enforce shapefile requirement of at least one field, to avoid writing invalid shapefiles [@Jonty] -- Fix Reader geo interface including DeletionFlag field in feature properties [@nnseva] -- Fix polygons not being auto closed, which was accidentally dropped -- Fix error for null geometries in feature geojson -- Misc docstring cleanup [@fiveham] + >>> shapes[3].shapeTypeName + 'POLYGON' -## 2.1.0 + * `bbox`: If the shape type contains multiple points this named tuple describes the + lower left (x,y) coordinate and upper right corner coordinate creating a + complete box around the points. If the shapeType is a + Null (shapeType == 0) then an AttributeError is raised. -### New Features: -- Added back read/write support for unicode field names. -- Improved Record representation -- More support for geojson on Reader, ShapeRecord, ShapeRecords, and shapes() + >>> # Get the bounding box of the 4th shape. + >>> shapes[3].bbox + BBox(xmin=-122.485792, ymin=37.786931, xmax=-122.446285, ymax=37.811019) + >>> # Round coordinates to 3 decimal places + >>> [f'{coord:.3f}' for coord in shapes[3].bbox] + ['-122.486', '37.787', '-122.446', '37.811'] -### Bug fixes: + * `parts`: Parts simply group collections of points into shapes. If the shape + record has multiple parts this attribute contains the index of the first + point of each part. If there is only one part then a list containing 0 is + returned. -- Fixed error when reading optional m-values -- Fixed Record attribute autocomplete in Python 3 -- Misc readme cleanup -## 2.0.0 + >>> shapes[3].parts + [0] -The newest version of PyShp, version 2.0 introduced some major new improvements. -A great thanks to all who have contributed code and raised issues, and for everyone's -patience and understanding during the transition period. -Some of the new changes are incompatible with previous versions. -Users of the previous version 1.x should therefore take note of the following changes -(Note: Some contributor attributions may be missing): + * `points_2D`/`points_3D`: The points_2D and points_3D attributes contain lists + of tuples containing (x,y) or (x,y,z) coordinates respectively for each + point in the shape. If no z data is available, z is set to 0 is used. -### Major Changes: + >>> len(shapes[3].points_2D) + 173 + >>> # Get the 8th point of the fourth shape + >>> # Truncate coordinates to 3 decimal places + >>> coords = shapes[3].points_2D[7] + >>> [f'{coord:.3f}' for coord in coords] + ['-122.471', '37.787'] -- Full support for unicode text, with custom encoding, and exception handling. - - Means that the Reader returns unicode, and the Writer accepts unicode. -- PyShp has been simplified to a pure input-output library using the Reader and Writer classes, dropping the Editor class. -- Switched to a new streaming approach when writing files, keeping memory-usage at a minimum: - - Specify filepath/destination and text encoding when creating the Writer. - - The file is written incrementally with each call to shape/record. - - Adding shapes is now done using dedicated methods for each shapetype. -- Reading shapefiles is now more convenient: - - Shapefiles can be opened using the context manager, and files are properly closed. - - Shapefiles can be iterated, have a length, and supports the geo interface. - - New ways of inspecting shapefile metadata by printing. [@megies] - - More convenient accessing of Record values as attributes. [@philippkraft] - - More convenient shape type name checking. [@megies] -- Add more support and documentation for MultiPatch 3D shapes. -- The Reader "elevation" and "measure" attributes now renamed "zbox" and "mbox", to make it clear they refer to the min/max values. -- Better documentation of previously unclear aspects, such as field types. +In most cases, however, if you need to do more than just type or bounds checking, you may want +to convert the geometry to the more human-readable [GeoJSON format](http://geojson.org), +where lines and polygons are grouped for you: -### Important Fixes: -- More reliable/robust: - - Fixed shapefile bbox error for empty or point type shapefiles. [@mcuprjak] - - Reading and writing Z and M type shapes is now more robust, fixing many errors, and has been added to the documentation. [@ShinNoNoir] - - Improved parsing of field value types, fixed errors and made more flexible. - - Fixed bug when writing shapefiles with datefield and date values earlier than 1900 [@megies] -- Fix some geo interface errors, including checking polygon directions. -- Bug fixes for reading from case sensitive file names, individual files separately, and from file-like objects. [@gastoneb, @kb003308, @erickskb] -- Enforce maximum field limit. [@mwtoews] + >>> s = sf.shape(0) + >>> geoj = s.__geo_interface__ + >>> geoj["type"] + 'MultiPolygon' +The results from the shapes() method similarly supports converting to GeoJSON: -# The Basics -Before doing anything you must import the library. + >>> shapes.__geo_interface__['type'] + 'GeometryCollection' +Note: In some cases, if the conversion from shapefile geometry to GeoJSON encountered any problems +or potential issues, a warning message will be displayed with information about the affected +geometry. To ignore or suppress these warnings, you can disable this behavior by setting the +module constant VERBOSE to False: - >>> import shapefile -The examples below will use a shapefile created from the U.S. Census Bureau -Blockgroups data set near San Francisco, CA and available in the git -repository of the PyShp GitHub site. + >>> shapefile.VERBOSE = False -## Reading Shapefiles -### The Reader Class +### Reading Records -#### Reading Shapefiles from Local Files +A record in a shapefile contains the attributes for each shape in the +collection of geometries. Records are stored in the dbf file. The link between +geometry and attributes is the foundation of all geographic information systems. +This critical link is implied by the order of shapes and corresponding records +in the shp geometry file and the dbf attribute file. -To read a shapefile create a new "Reader" object and pass it the name of an -existing shapefile. The shapefile format is actually a collection of three -files. You specify the base filename of the shapefile or the complete filename -of any of the shapefile component files. +The field names of a shapefile are available as soon as you read a shapefile. +You can call the "fields" attribute of the shapefile as a Python list. Each +field is a Python namedtuple (Field) with the following information: + * name: the name describing the data at this column index (a string). + * field_type: a FieldType determining the type of data at this column index. Names can be: + * "C": Characters, text. + * "N": Numbers, with or without decimals. + * "F": Floats (same as "N"). + * "L": Logical, for boolean True/False values. + * "D": Dates. + * "M": Memo, has no meaning within a GIS and is part of the xbase spec instead. + * size: Field length: the length of the data found at this column index. Older GIS + software may truncate this length to 8 or 11 characters for "Character" + fields. + * deci: Decimal length. The number of decimal places found in "Number" fields. - >>> sf = shapefile.Reader("tests/shapefiles/blockgroups") +A new field can be created as follows: -OR + >>> shapefile.Field("Population", "N", 10,0) + Field(name="Population", field_type=FieldType.N, size=10, decimal=0) +Using this method the conversion from string to enum is done automatically. - >>> sf = shapefile.Reader("tests/shapefiles/blockgroups.shp") +To see the fields for the Reader object above (sf) call the "fields" +attribute: -OR + >>> sf.fields + [Field(name="DeletionFlag", field_type=FieldType.C, size=1, decimal=0), Field(name="AREA", field_type=FieldType.N, size=18, decimal=5), Field(name="BKG_KEY", field_type=FieldType.C, size=12, decimal=0), Field(name="POP1990", field_type=FieldType.N, size=9, decimal=0), Field(name="POP90_SQMI", field_type=FieldType.N, size=10, decimal=1), Field(name="HOUSEHOLDS", field_type=FieldType.N, size=9, decimal=0), Field(name="MALES", field_type=FieldType.N, size=9, decimal=0), Field(name="FEMALES", field_type=FieldType.N, size=9, decimal=0), Field(name="WHITE", field_type=FieldType.N, size=9, decimal=0), Field(name="BLACK", field_type=FieldType.N, size=8, decimal=0), Field(name="AMERI_ES", field_type=FieldType.N, size=7, decimal=0), Field(name="ASIAN_PI", field_type=FieldType.N, size=8, decimal=0), Field(name="OTHER", field_type=FieldType.N, size=8, decimal=0), Field(name="HISPANIC", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_UNDER5", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_5_17", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_18_29", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_30_49", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_50_64", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_65_UP", field_type=FieldType.N, size=8, decimal=0), Field(name="NEVERMARRY", field_type=FieldType.N, size=8, decimal=0), Field(name="MARRIED", field_type=FieldType.N, size=9, decimal=0), Field(name="SEPARATED", field_type=FieldType.N, size=7, decimal=0), Field(name="WIDOWED", field_type=FieldType.N, size=8, decimal=0), Field(name="DIVORCED", field_type=FieldType.N, size=8, decimal=0), Field(name="HSEHLD_1_M", field_type=FieldType.N, size=8, decimal=0), Field(name="HSEHLD_1_F", field_type=FieldType.N, size=8, decimal=0), Field(name="MARHH_CHD", field_type=FieldType.N, size=8, decimal=0), Field(name="MARHH_NO_C", field_type=FieldType.N, size=8, decimal=0), Field(name="MHH_CHILD", field_type=FieldType.N, size=7, decimal=0), Field(name="FHH_CHILD", field_type=FieldType.N, size=7, decimal=0), Field(name="HSE_UNITS", field_type=FieldType.N, size=9, decimal=0), Field(name="VACANT", field_type=FieldType.N, size=7, decimal=0), Field(name="OWNER_OCC", field_type=FieldType.N, size=8, decimal=0), Field(name="RENTER_OCC", field_type=FieldType.N, size=8, decimal=0), Field(name="MEDIAN_VAL", field_type=FieldType.N, size=7, decimal=0), Field(name="MEDIANRENT", field_type=FieldType.N, size=4, decimal=0), Field(name="UNITS_1DET", field_type=FieldType.N, size=8, decimal=0), Field(name="UNITS_1ATT", field_type=FieldType.N, size=7, decimal=0), Field(name="UNITS2", field_type=FieldType.N, size=7, decimal=0), Field(name="UNITS3_9", field_type=FieldType.N, size=8, decimal=0), Field(name="UNITS10_49", field_type=FieldType.N, size=8, decimal=0), Field(name="UNITS50_UP", field_type=FieldType.N, size=8, decimal=0), Field(name="MOBILEHOME", field_type=FieldType.N, size=7, decimal=0)] - >>> sf = shapefile.Reader("tests/shapefiles/blockgroups.dbf") +The first field of a dbf file is always a 1-byte field called "DeletionFlag", +which indicates records that have been deleted but not removed. However, +since this flag is very rarely used, PyShp currently will return all records +regardless of their deletion flag, and the flag is also not included in the list of +record values. In other words, the DeletionFlag field has no real purpose, and +should in most cases be ignored. For instance, to get a list of all fieldnames: -OR any of the other 5+ formats which are potentially part of a shapefile. The -library does not care about file extensions. You can also specify that you only -want to read some of the file extensions through the use of keyword arguments: + >>> fieldnames = [f[0] for f in sf.fields[1:]] - >>> sf = shapefile.Reader(dbf="tests/shapefiles/blockgroups.dbf") +You can get a list of the shapefile's records by calling the records() method: -#### Reading Shapefiles from Zip Files -If your shapefile is wrapped inside a zip file, the library is able to handle that too, meaning you don't have to worry about unzipping the contents: + >>> records = sf.records() + >>> len(records) + 663 - >>> sf = shapefile.Reader("tests/shapefiles/blockgroups.zip") +To read a single record call the record() method with the record's index: -If the zip file contains multiple shapefiles, just specify which shapefile to read by additionally specifying the relative path after the ".zip" part: + >>> rec = sf.record(3) - >>> sf = shapefile.Reader("tests/shapefiles/blockgroups_multishapefile.zip/blockgroups2.shp") +Each record is a list-like Record object containing the values corresponding to each field in +the field list (except the DeletionFlag). A record's values can be accessed by positional indexing or slicing. +For example in the blockgroups shapefile the 2nd and 3rd fields are the blockgroup id +and the 1990 population count of that San Francisco blockgroup: -#### Reading Shapefiles from URLs -Finally, you can use all of the above methods to read shapefiles directly from the internet, by giving a url instead of a local path, e.g.: + >>> rec[1:3] + ['060750601001', 4715] +For simpler access, the fields of a record can also accessed via the name of the field, +either as a key or as an attribute name. The blockgroup id (BKG_KEY) of the blockgroups shapefile +can also be retrieved as: - >>> # from a zipped shapefile on website - >>> sf = shapefile.Reader("https://github.com/JamesParrott/PyShp_test_shapefile/raw/refs/heads/main/gadm41_ATA_shp.zip") - >>> # from a shapefile collection of files in a github repository - >>> sf = shapefile.Reader("https://pubs.usgs.gov/of/2000/of00-006/gisdata/coverage/airports.shp") + >>> rec['BKG_KEY'] + '060750601001' -This will automatically download the file(s) to a temporary location before reading, saving you a lot of time and repetitive boilerplate code when you just want quick access to some external data. + >>> rec.BKG_KEY + '060750601001' -#### Reading Shapefiles from File-Like Objects +The record values can be easily integrated with other programs by converting it to a field-value dictionary: -You can also load shapefiles from any Python file-like object using keyword -arguments to specify any of the three files. This feature is very powerful and -allows you to custom load shapefiles from arbitrary storage formats, such as a protected url or zip file, a serialized object, or in some cases a database. + >>> dct = rec.as_dict() + >>> sorted(dct.items()) + [('AGE_18_29', 1467), ('AGE_30_49', 1681), ('AGE_50_64', 92), ('AGE_5_17', 848), ('AGE_65_UP', 30), ('AGE_UNDER5', 597), ('AMERI_ES', 6), ('AREA', 2.34385), ('ASIAN_PI', 452), ('BKG_KEY', '060750601001'), ('BLACK', 1007), ('DIVORCED', 149), ('FEMALES', 2095), ('FHH_CHILD', 16), ('HISPANIC', 416), ('HOUSEHOLDS', 1195), ('HSEHLD_1_F', 40), ('HSEHLD_1_M', 22), ('HSE_UNITS', 1258), ('MALES', 2620), ('MARHH_CHD', 79), ('MARHH_NO_C', 958), ('MARRIED', 2021), ('MEDIANRENT', 739), ('MEDIAN_VAL', 337500), ('MHH_CHILD', 0), ('MOBILEHOME', 0), ('NEVERMARRY', 703), ('OTHER', 288), ('OWNER_OCC', 66), ('POP1990', 4715), ('POP90_SQMI', 2011.6), ('RENTER_OCC', 3733), ('SEPARATED', 49), ('UNITS10_49', 49), ('UNITS2', 160), ('UNITS3_9', 672), ('UNITS50_UP', 0), ('UNITS_1ATT', 302), ('UNITS_1DET', 43), ('VACANT', 93), ('WHITE', 2962), ('WIDOWED', 37)] - >>> myshp = open("tests/shapefiles/blockgroups.shp", "rb") - >>> mydbf = open("tests/shapefiles/blockgroups.dbf", "rb") - >>> r = shapefile.Reader(shp=myshp, dbf=mydbf) +If at a later point you need to check the record's index position in the original +shapefile, you can do this through the "oid" attribute: -Notice in the examples above the shx file is never used. The shx file is a -very simple fixed-record index for the variable-length records in the shp -file. This file is optional for reading. If it's available PyShp will use the -shx file to access shape records a little faster but will do just fine without -it. -#### Reading Shapefiles Using the Context Manager + >>> rec.oid + 3 -The "Reader" class can be used as a context manager, to ensure open file -objects are properly closed when done reading the data: +### Reading Geometry and Records Simultaneously - >>> with shapefile.Reader("tests/shapefiles/blockgroups.shp") as shp: - ... print(shp) - shapefile Reader - 663 shapes (type 'POLYGON') - 663 records (44 fields) +You may want to examine both the geometry and the attributes for a record at +the same time. The shapeRecord() and shapeRecords() method let you do just +that. -#### Reading Shapefile Meta-Data +Calling the shapeRecords() method will return the geometry and attributes for +all shapes as a list of ShapeRecord objects. Each ShapeRecord instance has a +"shape" and "record" attribute. The shape attribute is a Shape object as +discussed in the first section "Reading Geometry". The record attribute is a +list-like object containing field values as demonstrated in the "Reading Records" section. -Shapefiles have a number of attributes for inspecting the file contents. -A shapefile is a container for a specific type of geometry, and this can be checked using the -shapeType attribute. + >>> shapeRecs = sf.shapeRecords() - >>> sf = shapefile.Reader("tests/shapefiles/blockgroups.dbf") - >>> sf.shapeType - 5 +Let's read the blockgroup key and the population for the 4th blockgroup: -Shape types are represented by numbers between 0 and 31 as defined by the -shapefile specification and listed below. It is important to note that the numbering system has -several reserved numbers that have not been used yet, therefore the numbers of -the existing shape types are not sequential: -- NULL = 0 -- POINT = 1 -- POLYLINE = 3 -- POLYGON = 5 -- MULTIPOINT = 8 -- POINTZ = 11 -- POLYLINEZ = 13 -- POLYGONZ = 15 -- MULTIPOINTZ = 18 -- POINTM = 21 -- POLYLINEM = 23 -- POLYGONM = 25 -- MULTIPOINTM = 28 -- MULTIPATCH = 31 + >>> shapeRecs[3].record[1:3] + ['060750601001', 4715] -Based on this we can see that our blockgroups shapefile contains -Polygon type shapes. The shape types are also defined as constants in -the shapefile module, so that we can compare types more intuitively: +The results from the shapeRecords() method is a list-like object that can be easily converted +to GeoJSON through the _\_geo_interface\_\_: - >>> sf.shapeType == shapefile.POLYGON - True + >>> shapeRecs.__geo_interface__['type'] + 'FeatureCollection' -For convenience, you can also get the name of the shape type as a string: +The shapeRecord() method reads a single shape/record pair at the specified index. +To get the 4th shape record from the blockgroups shapefile use the third index: - >>> sf.shapeTypeName == 'POLYGON' - True + >>> shapeRec = sf.shapeRecord(3) + >>> shapeRec.record[1:3] + ['060750601001', 4715] -Other pieces of meta-data that we can check include the number of features -and the bounding box area the shapefile covers: +Each individual shape record also supports the _\_geo_interface\_\_ to convert it to a GeoJSON feature: - >>> len(sf) - 663 - >>> sf.bbox - BBox(xmin=-122.515048, ymin=37.652916, xmax=-122.327622, ymax=37.863433) + >>> shapeRec.__geo_interface__['type'] + 'Feature' -Finally, if you would prefer to work with the entire shapefile in a different -format, you can convert all of it to a GeoJSON dictionary, although you may lose -some information in the process, such as z- and m-values: +## Writing Shapefiles - >>> sf.__geo_interface__['type'] - 'FeatureCollection' +### The Writer Class -### Reading Geometry +PyShp tries to be as flexible as possible when writing shapefiles while +maintaining some degree of automatic validation to make sure you don't +accidentally write an invalid file. -A shapefile's geometry is the collection of points or shapes made from -vertices and implied arcs representing physical locations. All types of -shapefiles just store points. The metadata about the points determine how they -are handled by software. +PyShp can write just one of the component files such as the shp or dbf file +without writing the others. So in addition to being a complete shapefile +library, it can also be used as a basic dbf (xbase) library. Dbf files are a +common database format which are often useful as a standalone simple database +format. And even shp files occasionally have uses as a standalone format. Some +web-based GIS systems use an user-uploaded shp file to specify an area of +interest. Many precision agriculture chemical field sprayers also use the shp +format as a control file for the sprayer system (usually in combination with +custom database file formats). -You can get a list of the shapefile's geometry by calling the shapes() -method. +#### Writing Shapefiles to Local Files +To create a shapefile you begin by initiating a new Writer instance, passing it +the file path and name to save to: - >>> shapes = sf.shapes() -The shapes method returns a list of Shape objects describing the geometry of -each shape record. + >>> w = shapefile.Writer('tests/shapefiles/test/testfile') + >>> w.field('field1', 'C') +File extensions are optional when reading or writing shapefiles. If you specify +them PyShp ignores them anyway. When you save files you can specify a base +file name that is used for all three file types. Or you can specify a name for +one or more file types: - >>> len(shapes) - 663 -To read a single shape by calling its index use the shape() method. The index -is the shape's count from 0. So to read the 8th shape record you would use its -index which is 7. + >>> w = shapefile.Writer(dbf='tests/shapefiles/test/onlydbf.dbf') + >>> w.field('field1', 'C') +In that case, any file types not assigned will not +save and only file types with file names will be saved. - >>> s = sf.shape(7) - >>> s - Polygon #7 +#### Writing Shapefiles to File-Like Objects +Just as you can read shapefiles from python file-like objects you can also +write to them: - >>> # Read the bbox of the 8th shape to verify - >>> s.bbox - BBox(xmin=-122.449637, ymin=37.80149, xmax=-122.442109, ymax=37.807958) - >>> # Round coordinates to 3 decimal places - >>> [f'{coord:.3f}' for coord in s.bbox] - ['-122.450', '37.801', '-122.442', '37.808'] -Each shape record (except Points) contains the following attributes. Records of -shapeType Point do not have a bounding box 'bbox'. + >>> try: + ... from StringIO import StringIO + ... except ImportError: + ... from io import BytesIO as StringIO + >>> shp = StringIO() + >>> shx = StringIO() + >>> dbf = StringIO() + >>> w = shapefile.Writer(shp=shp, shx=shx, dbf=dbf) + >>> w.field('field1', 'C') + >>> w.record() + >>> w.null() + >>> w.close() - >>> for name in dir(shapes[3]): - ... if not name.startswith('_'): - ... name - 'bbox' - 'from_byte_stream' - 'oid' - 'parts' - 'points' - 'points_2D' - 'points_3D' - 'shapeType' - 'shapeTypeName' - 'write_to_byte_stream' + >>> # To read back the files you could call the "StringIO.getvalue()" method later. + >>> assert shp.getvalue() + >>> assert shx.getvalue() + >>> assert dbf.getvalue() - * `oid`: The shape's index position in the original shapefile. + >>> # In fact, you can read directly from them using the Reader + >>> r = shapefile.Reader(shp=shp, shx=shx, dbf=dbf) + >>> len(r) + 1 - >>> shapes[3].oid - 3 - * `shapeType`: an integer representing the type of shape as defined by the - shapefile specification. +#### Writing Shapefiles Using the Context Manager +The "Writer" class automatically closes the open files and writes the final headers once it is garbage collected. +In case of a crash and to make the code more readable, it is nevertheless recommended +you do this manually by calling the "close()" method: - >>> shapes[3].shapeType - 5 - * `shapeTypeName`: a string representation of the type of shape as defined by shapeType. Read-only. + >>> w.close() +Alternatively, you can also use the "Writer" class as a context manager, to ensure open file +objects are properly closed and final headers written once you exit the with-clause: - >>> shapes[3].shapeTypeName - 'POLYGON' - * `bbox`: If the shape type contains multiple points this named tuple describes the - lower left (x,y) coordinate and upper right corner coordinate creating a - complete box around the points. If the shapeType is a - Null (shapeType == 0) then an AttributeError is raised. + >>> with shapefile.Writer("tests/shapefiles/test/contextwriter") as w: + ... w.field('field1', 'C') + ... pass +#### Setting the Shape Type - >>> # Get the bounding box of the 4th shape. - >>> shapes[3].bbox - BBox(xmin=-122.485792, ymin=37.786931, xmax=-122.446285, ymax=37.811019) - >>> # Round coordinates to 3 decimal places - >>> [f'{coord:.3f}' for coord in shapes[3].bbox] - ['-122.486', '37.787', '-122.446', '37.811'] - - * `parts`: Parts simply group collections of points into shapes. If the shape - record has multiple parts this attribute contains the index of the first - point of each part. If there is only one part then a list containing 0 is - returned. +The shape type defines the type of geometry contained in the shapefile. All of +the shapes must match the shape type setting. +There are three ways to set the shape type: + * Set it when creating the class instance. + * Set it by assigning a value to an existing class instance. + * Set it automatically to the type of the first non-null shape by saving the shapefile. - >>> shapes[3].parts - [0] +To manually set the shape type for a Writer object when creating the Writer: - * `points_2D`/`points_3D`: The points_2D and points_3D attributes contain lists - of tuples containing (x,y) or (x,y,z) coordinates respectively for each - point in the shape. If no z data is available, z is set to 0 is used. - >>> len(shapes[3].points_2D) - 173 - >>> # Get the 8th point of the fourth shape - >>> # Truncate coordinates to 3 decimal places - >>> coords = shapes[3].points_2D[7] - >>> [f'{coord:.3f}' for coord in coords] - ['-122.471', '37.787'] + >>> w = shapefile.Writer('tests/shapefiles/test/shapetype', shapeType=3) + >>> w.field('field1', 'C') -In most cases, however, if you need to do more than just type or bounds checking, you may want -to convert the geometry to the more human-readable [GeoJSON format](http://geojson.org), -where lines and polygons are grouped for you: + >>> w.shapeType + 3 +OR you can set it after the Writer is created: - >>> s = sf.shape(0) - >>> geoj = s.__geo_interface__ - >>> geoj["type"] - 'MultiPolygon' -The results from the shapes() method similarly supports converting to GeoJSON: + >>> w.shapeType = 1 + >>> w.shapeType + 1 - >>> shapes.__geo_interface__['type'] - 'GeometryCollection' -Note: In some cases, if the conversion from shapefile geometry to GeoJSON encountered any problems -or potential issues, a warning message will be displayed with information about the affected -geometry. To ignore or suppress these warnings, you can disable this behavior by setting the -module constant VERBOSE to False: +### Adding Records +Before you can add records you must first create the fields that define what types of +values will go into each attribute. - >>> shapefile.VERBOSE = False +There are several different field types, all of which support storing None values as NULL. +Text fields are created using the 'C' type, and the third 'size' argument can be customized to the expected +length of text values to save space: -### Reading Records -A record in a shapefile contains the attributes for each shape in the -collection of geometries. Records are stored in the dbf file. The link between -geometry and attributes is the foundation of all geographic information systems. -This critical link is implied by the order of shapes and corresponding records -in the shp geometry file and the dbf attribute file. + >>> w = shapefile.Writer('tests/shapefiles/test/dtype') + >>> w.field('TEXT', 'C') + >>> w.field('SHORT_TEXT', 'C', size=5) + >>> w.field('LONG_TEXT', 'C', size=250) + >>> w.null() + >>> w.record('Hello', 'World', 'World'*50) + >>> w.close() -The field names of a shapefile are available as soon as you read a shapefile. -You can call the "fields" attribute of the shapefile as a Python list. Each -field is a Python namedtuple (Field) with the following information: + >>> r = shapefile.Reader('tests/shapefiles/test/dtype') + >>> assert r.record(0) == ['Hello', 'World', 'World'*50] + >>> r.close() - * name: the name describing the data at this column index (a string). - * field_type: a FieldType determining the type of data at this column index. Names can be: - * "C": Characters, text. - * "N": Numbers, with or without decimals. - * "F": Floats (same as "N"). - * "L": Logical, for boolean True/False values. - * "D": Dates. - * "M": Memo, has no meaning within a GIS and is part of the xbase spec instead. - * size: Field length: the length of the data found at this column index. Older GIS - software may truncate this length to 8 or 11 characters for "Character" - fields. - * deci: Decimal length. The number of decimal places found in "Number" fields. +Date fields are created using the 'D' type, and can be created using either +date objects, lists, or a YYYYMMDD formatted string. +Field length or decimal have no impact on this type: -A new field can be created as follows: - >>> shapefile.Field("Population", "N", 10,0) - Field(name="Population", field_type=FieldType.N, size=10, decimal=0) + >>> from datetime import date + >>> w = shapefile.Writer('tests/shapefiles/test/dtype') + >>> w.field('DATE', 'D') + >>> w.null() + >>> w.null() + >>> w.null() + >>> w.null() + >>> w.record(date(1898,1,30)) + >>> w.record([1998,1,30]) + >>> w.record('19980130') + >>> w.record(None) + >>> w.close() -Using this method the conversion from string to enum is done automatically. + >>> r = shapefile.Reader('tests/shapefiles/test/dtype') + >>> assert r.record(0) == [date(1898,1,30)] + >>> assert r.record(1) == [date(1998,1,30)] + >>> assert r.record(2) == [date(1998,1,30)] + >>> assert r.record(3) == [None] + >>> r.close() -To see the fields for the Reader object above (sf) call the "fields" -attribute: +Numeric fields are created using the 'N' type (or the 'F' type, which is exactly the same). +By default the fourth decimal argument is set to zero, essentially creating an integer field. +To store floats you must set the decimal argument to the precision of your choice. +To store very large numbers you must increase the field length size to the total number of digits +(including comma and minus). - >>> sf.fields - [Field(name="DeletionFlag", field_type=FieldType.C, size=1, decimal=0), Field(name="AREA", field_type=FieldType.N, size=18, decimal=5), Field(name="BKG_KEY", field_type=FieldType.C, size=12, decimal=0), Field(name="POP1990", field_type=FieldType.N, size=9, decimal=0), Field(name="POP90_SQMI", field_type=FieldType.N, size=10, decimal=1), Field(name="HOUSEHOLDS", field_type=FieldType.N, size=9, decimal=0), Field(name="MALES", field_type=FieldType.N, size=9, decimal=0), Field(name="FEMALES", field_type=FieldType.N, size=9, decimal=0), Field(name="WHITE", field_type=FieldType.N, size=9, decimal=0), Field(name="BLACK", field_type=FieldType.N, size=8, decimal=0), Field(name="AMERI_ES", field_type=FieldType.N, size=7, decimal=0), Field(name="ASIAN_PI", field_type=FieldType.N, size=8, decimal=0), Field(name="OTHER", field_type=FieldType.N, size=8, decimal=0), Field(name="HISPANIC", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_UNDER5", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_5_17", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_18_29", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_30_49", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_50_64", field_type=FieldType.N, size=8, decimal=0), Field(name="AGE_65_UP", field_type=FieldType.N, size=8, decimal=0), Field(name="NEVERMARRY", field_type=FieldType.N, size=8, decimal=0), Field(name="MARRIED", field_type=FieldType.N, size=9, decimal=0), Field(name="SEPARATED", field_type=FieldType.N, size=7, decimal=0), Field(name="WIDOWED", field_type=FieldType.N, size=8, decimal=0), Field(name="DIVORCED", field_type=FieldType.N, size=8, decimal=0), Field(name="HSEHLD_1_M", field_type=FieldType.N, size=8, decimal=0), Field(name="HSEHLD_1_F", field_type=FieldType.N, size=8, decimal=0), Field(name="MARHH_CHD", field_type=FieldType.N, size=8, decimal=0), Field(name="MARHH_NO_C", field_type=FieldType.N, size=8, decimal=0), Field(name="MHH_CHILD", field_type=FieldType.N, size=7, decimal=0), Field(name="FHH_CHILD", field_type=FieldType.N, size=7, decimal=0), Field(name="HSE_UNITS", field_type=FieldType.N, size=9, decimal=0), Field(name="VACANT", field_type=FieldType.N, size=7, decimal=0), Field(name="OWNER_OCC", field_type=FieldType.N, size=8, decimal=0), Field(name="RENTER_OCC", field_type=FieldType.N, size=8, decimal=0), Field(name="MEDIAN_VAL", field_type=FieldType.N, size=7, decimal=0), Field(name="MEDIANRENT", field_type=FieldType.N, size=4, decimal=0), Field(name="UNITS_1DET", field_type=FieldType.N, size=8, decimal=0), Field(name="UNITS_1ATT", field_type=FieldType.N, size=7, decimal=0), Field(name="UNITS2", field_type=FieldType.N, size=7, decimal=0), Field(name="UNITS3_9", field_type=FieldType.N, size=8, decimal=0), Field(name="UNITS10_49", field_type=FieldType.N, size=8, decimal=0), Field(name="UNITS50_UP", field_type=FieldType.N, size=8, decimal=0), Field(name="MOBILEHOME", field_type=FieldType.N, size=7, decimal=0)] + >>> w = shapefile.Writer('tests/shapefiles/test/dtype') + >>> w.field('INT', 'N') + >>> w.field('LOWPREC', 'N', decimal=2) + >>> w.field('MEDPREC', 'N', decimal=10) + >>> w.field('HIGHPREC', 'N', decimal=30) + >>> w.field('FTYPE', 'F', decimal=10) + >>> w.field('LARGENR', 'N', 101) + >>> nr = 1.3217328 + >>> w.null() + >>> w.null() + >>> w.record(INT=nr, LOWPREC=nr, MEDPREC=nr, HIGHPREC=-3.2302e-25, FTYPE=nr, LARGENR=int(nr)*10**100) + >>> w.record(None, None, None, None, None, None) + >>> w.close() -The first field of a dbf file is always a 1-byte field called "DeletionFlag", -which indicates records that have been deleted but not removed. However, -since this flag is very rarely used, PyShp currently will return all records -regardless of their deletion flag, and the flag is also not included in the list of -record values. In other words, the DeletionFlag field has no real purpose, and -should in most cases be ignored. For instance, to get a list of all fieldnames: + >>> r = shapefile.Reader('tests/shapefiles/test/dtype') + >>> assert r.record(0) == [1, 1.32, 1.3217328, -3.2302e-25, 1.3217328, 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] + >>> assert r.record(1) == [None, None, None, None, None, None] + >>> r.close() - >>> fieldnames = [f[0] for f in sf.fields[1:]] +Finally, we can create boolean fields by setting the type to 'L'. +This field can take True or False values, or 1 (True) or 0 (False). +None is interpreted as missing. -You can get a list of the shapefile's records by calling the records() method: + >>> w = shapefile.Writer('tests/shapefiles/test/dtype') + >>> w.field('BOOLEAN', 'L') + >>> w.null() + >>> w.null() + >>> w.null() + >>> w.null() + >>> w.null() + >>> w.null() + >>> w.record(True) + >>> w.record(1) + >>> w.record(False) + >>> w.record(0) + >>> w.record(None) + >>> w.record("Nonsense") + >>> w.close() - >>> records = sf.records() + >>> r = shapefile.Reader('tests/shapefiles/test/dtype') + >>> r.record(0) + Record #0: [True] + >>> r.record(1) + Record #1: [True] + >>> r.record(2) + Record #2: [False] + >>> r.record(3) + Record #3: [False] + >>> r.record(4) + Record #4: [None] + >>> r.record(5) + Record #5: [None] + >>> r.close() - >>> len(records) - 663 +You can also add attributes using keyword arguments where the keys are field names. -To read a single record call the record() method with the record's index: + >>> w = shapefile.Writer('tests/shapefiles/test/dtype') + >>> w.field('FIRST_FLD','C', 40) + >>> w.field('SECOND_FLD','C', 40) + >>> w.null() + >>> w.null() + >>> w.record('First', 'Line') + >>> w.record(FIRST_FLD='First', SECOND_FLD='Line') + >>> w.close() - >>> rec = sf.record(3) +### Adding Geometry -Each record is a list-like Record object containing the values corresponding to each field in -the field list (except the DeletionFlag). A record's values can be accessed by positional indexing or slicing. -For example in the blockgroups shapefile the 2nd and 3rd fields are the blockgroup id -and the 1990 population count of that San Francisco blockgroup: +Geometry is added using one of several convenience methods. The "null" method is used +for null shapes, "point" is used for point shapes, "multipoint" is used for multipoint shapes, "line" for lines, +"poly" for polygons. +**Adding a Null shape** - >>> rec[1:3] - ['060750601001', 4715] +A shapefile may contain some records for which geometry is not available, and may be set using the "null" method. +Because Null shape types (shape type 0) have no geometry the "null" method is called without any arguments. -For simpler access, the fields of a record can also accessed via the name of the field, -either as a key or as an attribute name. The blockgroup id (BKG_KEY) of the blockgroups shapefile -can also be retrieved as: + >>> w = shapefile.Writer('tests/shapefiles/test/null') + >>> w.field('name', 'C') - >>> rec['BKG_KEY'] - '060750601001' + >>> w.null() + >>> w.record('nullgeom') - >>> rec.BKG_KEY - '060750601001' + >>> w.close() -The record values can be easily integrated with other programs by converting it to a field-value dictionary: +**Adding a Point shape** +Point shapes are added using the "point" method. A point is specified by an x and +y value. - >>> dct = rec.as_dict() - >>> sorted(dct.items()) - [('AGE_18_29', 1467), ('AGE_30_49', 1681), ('AGE_50_64', 92), ('AGE_5_17', 848), ('AGE_65_UP', 30), ('AGE_UNDER5', 597), ('AMERI_ES', 6), ('AREA', 2.34385), ('ASIAN_PI', 452), ('BKG_KEY', '060750601001'), ('BLACK', 1007), ('DIVORCED', 149), ('FEMALES', 2095), ('FHH_CHILD', 16), ('HISPANIC', 416), ('HOUSEHOLDS', 1195), ('HSEHLD_1_F', 40), ('HSEHLD_1_M', 22), ('HSE_UNITS', 1258), ('MALES', 2620), ('MARHH_CHD', 79), ('MARHH_NO_C', 958), ('MARRIED', 2021), ('MEDIANRENT', 739), ('MEDIAN_VAL', 337500), ('MHH_CHILD', 0), ('MOBILEHOME', 0), ('NEVERMARRY', 703), ('OTHER', 288), ('OWNER_OCC', 66), ('POP1990', 4715), ('POP90_SQMI', 2011.6), ('RENTER_OCC', 3733), ('SEPARATED', 49), ('UNITS10_49', 49), ('UNITS2', 160), ('UNITS3_9', 672), ('UNITS50_UP', 0), ('UNITS_1ATT', 302), ('UNITS_1DET', 43), ('VACANT', 93), ('WHITE', 2962), ('WIDOWED', 37)] -If at a later point you need to check the record's index position in the original -shapefile, you can do this through the "oid" attribute: + >>> w = shapefile.Writer('tests/shapefiles/test/point') + >>> w.field('name', 'C') + >>> w.point(122, 37) + >>> w.record('point1') - >>> rec.oid - 3 + >>> w.close() -### Reading Geometry and Records Simultaneously +**Adding a MultiPoint shape** -You may want to examine both the geometry and the attributes for a record at -the same time. The shapeRecord() and shapeRecords() method let you do just -that. +If your point data allows for the possibility of multiple points per feature, use "multipoint" instead. +These are specified as a list of xy point coordinates. -Calling the shapeRecords() method will return the geometry and attributes for -all shapes as a list of ShapeRecord objects. Each ShapeRecord instance has a -"shape" and "record" attribute. The shape attribute is a Shape object as -discussed in the first section "Reading Geometry". The record attribute is a -list-like object containing field values as demonstrated in the "Reading Records" section. + >>> w = shapefile.Writer('tests/shapefiles/test/multipoint') + >>> w.field('name', 'C') - >>> shapeRecs = sf.shapeRecords() + >>> w.multipoint([[122,37], [124,32]]) + >>> w.record('multipoint1') -Let's read the blockgroup key and the population for the 4th blockgroup: + >>> w.close() +**Adding a LineString shape** - >>> shapeRecs[3].record[1:3] - ['060750601001', 4715] +For LineString shapefiles, each shape is given as a list of one or more linear features. +Each of the linear features must have at least two points. -The results from the shapeRecords() method is a list-like object that can be easily converted -to GeoJSON through the _\_geo_interface\_\_: + >>> w = shapefile.Writer('tests/shapefiles/test/line') + >>> w.field('name', 'C') - >>> shapeRecs.__geo_interface__['type'] - 'FeatureCollection' + >>> w.line([ + ... [[1,5],[5,5],[5,1],[3,3],[1,1]], # line 1 + ... [[3,2],[2,6]] # line 2 + ... ]) -The shapeRecord() method reads a single shape/record pair at the specified index. -To get the 4th shape record from the blockgroups shapefile use the third index: + >>> w.record('linestring1') + >>> w.close() - >>> shapeRec = sf.shapeRecord(3) - >>> shapeRec.record[1:3] - ['060750601001', 4715] +**Adding a Polygon shape** -Each individual shape record also supports the _\_geo_interface\_\_ to convert it to a GeoJSON feature: +Similarly to LineString, Polygon shapes consist of multiple polygons, and must be given as a list of polygons. +The main difference is that polygons must have at least 4 points and the last point must be the same as the first. +It's also okay if you forget to repeat the first point at the end; PyShp automatically checks and closes the polygons +if you don't. +It's important to note that for Polygon shapefiles, your polygon coordinates must be ordered in a clockwise direction. +If any of the polygons have holes, then the hole polygon coordinates must be ordered in a counterclockwise direction. +The direction of your polygons determines how shapefile readers will distinguish between polygon outlines and holes. - >>> shapeRec.__geo_interface__['type'] - 'Feature' + >>> w = shapefile.Writer('tests/shapefiles/test/polygon') + >>> w.field('name', 'C') -## Writing Shapefiles + >>> w.poly([ + ... [[113,24], [112,32], [117,36], [122,37], [118,20]], # poly 1 + ... [[116,29],[116,26],[119,29],[119,32]], # hole 1 + ... [[15,2], [17,6], [22,7]] # poly 2 + ... ]) + >>> w.record('polygon1') -### The Writer Class + >>> w.close() -PyShp tries to be as flexible as possible when writing shapefiles while -maintaining some degree of automatic validation to make sure you don't -accidentally write an invalid file. +**Adding from an existing Shape object** -PyShp can write just one of the component files such as the shp or dbf file -without writing the others. So in addition to being a complete shapefile -library, it can also be used as a basic dbf (xbase) library. Dbf files are a -common database format which are often useful as a standalone simple database -format. And even shp files occasionally have uses as a standalone format. Some -web-based GIS systems use an user-uploaded shp file to specify an area of -interest. Many precision agriculture chemical field sprayers also use the shp -format as a control file for the sprayer system (usually in combination with -custom database file formats). +Finally, geometry can be added by passing an existing "Shape" object to the "shape" method. +You can also pass it any GeoJSON dictionary or _\_geo_interface\_\_ compatible object. +This can be particularly useful for copying from one file to another: -#### Writing Shapefiles to Local Files -To create a shapefile you begin by initiating a new Writer instance, passing it -the file path and name to save to: + >>> r = shapefile.Reader('tests/shapefiles/test/polygon') + + >>> w = shapefile.Writer('tests/shapefiles/test/copy') + >>> w.fields = r.fields[1:] # skip first deletion field + >>> # adding existing Shape objects + >>> for shaperec in r.iterShapeRecords(): + ... w.record(*shaperec.record) + ... w.shape(shaperec.shape) - >>> w = shapefile.Writer('tests/shapefiles/test/testfile') - >>> w.field('field1', 'C') + >>> # or GeoJSON dicts + >>> for shaperec in r.iterShapeRecords(): + ... w.record(*shaperec.record) + ... w.shape(shaperec.shape.__geo_interface__) -File extensions are optional when reading or writing shapefiles. If you specify -them PyShp ignores them anyway. When you save files you can specify a base -file name that is used for all three file types. Or you can specify a name for -one or more file types: + >>> r.close() + >>> w.close() - >>> w = shapefile.Writer(dbf='tests/shapefiles/test/onlydbf.dbf') - >>> w.field('field1', 'C') +### Geometry and Record Balancing -In that case, any file types not assigned will not -save and only file types with file names will be saved. +Because every shape must have a corresponding record it is critical that the +number of records equals the number of shapes to create a valid shapefile. You +must take care to add records and shapes in the same order so that the record +data lines up with the geometry data. For example: -#### Writing Shapefiles to File-Like Objects -Just as you can read shapefiles from python file-like objects you can also -write to them: + >>> w = shapefile.Writer('tests/shapefiles/test/balancing', shapeType=shapefile.POINT) + >>> w.field("field1", "C") + >>> w.field("field2", "C") + >>> w.record("row", "one") + >>> w.point(1, 1) - >>> try: - ... from StringIO import StringIO - ... except ImportError: - ... from io import BytesIO as StringIO - >>> shp = StringIO() - >>> shx = StringIO() - >>> dbf = StringIO() - >>> w = shapefile.Writer(shp=shp, shx=shx, dbf=dbf) - >>> w.field('field1', 'C') - >>> w.record() - >>> w.null() - >>> w.close() + >>> w.record("row", "two") + >>> w.point(2, 2) - >>> # To read back the files you could call the "StringIO.getvalue()" method later. - >>> assert shp.getvalue() - >>> assert shx.getvalue() - >>> assert dbf.getvalue() +To help prevent accidental misalignment PyShp has an "auto balance" feature to +make sure when you add either a shape or a record the two sides of the +equation line up. This way if you forget to update an entry the +shapefile will still be valid and handled correctly by most shapefile +software. Autobalancing is NOT turned on by default. To activate it set +the attribute autoBalance to 1 or True: - >>> # In fact, you can read directly from them using the Reader - >>> r = shapefile.Reader(shp=shp, shx=shx, dbf=dbf) - >>> len(r) - 1 + >>> w.autoBalance = 1 + >>> w.record("row", "three") + >>> w.record("row", "four") + >>> w.point(4, 4) + + >>> w.recNum == w.shpNum + True +You also have the option of manually calling the balance() method at any time +to ensure the other side is up to date. When balancing is used +null shapes are created on the geometry side or records +with a value of "NULL" for each field is created on the attribute side. +This gives you flexibility in how you build the shapefile. +You can create all of the shapes and then create all of the records or vice versa. -#### Writing Shapefiles Using the Context Manager -The "Writer" class automatically closes the open files and writes the final headers once it is garbage collected. -In case of a crash and to make the code more readable, it is nevertheless recommended -you do this manually by calling the "close()" method: + >>> w.autoBalance = 0 + >>> w.record("row", "five") + >>> w.record("row", "six") + >>> w.record("row", "seven") + >>> w.point(5, 5) + >>> w.point(6, 6) + >>> w.balance() + >>> w.recNum == w.shpNum + True - >>> w.close() +If you do not use the autoBalance() or balance() method and forget to manually +balance the geometry and attributes the shapefile will be viewed as corrupt by +most shapefile software. -Alternatively, you can also use the "Writer" class as a context manager, to ensure open file -objects are properly closed and final headers written once you exit the with-clause: +### Writing .prj files +A .prj file, or projection file, is a simple text file that stores a shapefile's map projection and coordinate reference system to help mapping software properly locate the geometry on a map. If you don't have one, you may get confusing errors when you try and use the shapefile you created. The GIS software may complain that it doesn't know the shapefile's projection and refuse to accept it, it may assume the shapefile is the same projection as the rest of your GIS project and put it in the wrong place, or it might assume the coordinates are an offset in meters from latitude and longitude 0,0 which will put your data in the middle of the ocean near Africa. The text in the .prj file is a [Well-Known-Text (WKT) projection string](https://en.wikipedia.org/wiki/Well-known_text_representation_of_coordinate_reference_systems). Projection strings can be quite long so they are often referenced using numeric codes call EPSG codes. The .prj file must have the same base name as your shapefile. So for example if you have a shapefile named "myPoints.shp", the .prj file must be named "myPoints.prj". +If you're using the same projection over and over, the following is a simple way to create the .prj file assuming your base filename is stored in a variable called "filename": - >>> with shapefile.Writer("tests/shapefiles/test/contextwriter") as w: - ... w.field('field1', 'C') - ... pass +``` + with open("{}.prj".format(filename), "w") as prj: + wkt = 'GEOGCS["WGS 84",' + wkt += 'DATUM["WGS_1984",' + wkt += 'SPHEROID["WGS 84",6378137,298.257223563]]' + wkt += ',PRIMEM["Greenwich",0],' + wkt += 'UNIT["degree",0.0174532925199433]]' + prj.write(wkt) +``` -#### Setting the Shape Type +If you need to dynamically fetch WKT projection strings, you can use the pure Python [PyCRS](https://github.com/karimbahgat/PyCRS) module which has a number of useful features. -The shape type defines the type of geometry contained in the shapefile. All of -the shapes must match the shape type setting. +# Advanced Use -There are three ways to set the shape type: - * Set it when creating the class instance. - * Set it by assigning a value to an existing class instance. - * Set it automatically to the type of the first non-null shape by saving the shapefile. +## Common Errors and Fixes -To manually set the shape type for a Writer object when creating the Writer: +Below we list some commonly encountered errors and ways to fix them. +### Warnings and Logging - >>> w = shapefile.Writer('tests/shapefiles/test/shapetype', shapeType=3) - >>> w.field('field1', 'C') +By default, PyShp chooses to be transparent and provide the user with logging information and warnings about non-critical issues when reading or writing shapefiles. This behavior is controlled by the module constant `VERBOSE` (which defaults to True). If you would rather suppress this information, you can simply set this to False: - >>> w.shapeType - 3 -OR you can set it after the Writer is created: + >>> shapefile.VERBOSE = False +All logging happens under the namespace `shapefile`. So another way to suppress all PyShp warnings would be to alter the logging behavior for that namespace: - >>> w.shapeType = 1 - >>> w.shapeType - 1 + >>> import logging + >>> logging.getLogger('shapefile').setLevel(logging.ERROR) +### Shapefile Encoding Errors -### Adding Records +PyShp supports reading and writing shapefiles in any language or character encoding, and provides several options for decoding and encoding text. +Most shapefiles are written in UTF-8 encoding, PyShp's default encoding, so in most cases you don't have to specify the encoding. +#### Encoding errors +If you encounter an encoding error when reading a shapefile, this means the shapefile was likely written in a non-utf8 encoding. +For instance, when working with English language shapefiles, a common reason for encoding errors is that the shapefile was written in Latin-1 encoding. +#### Recommendation +*We recommend avoiding writing Shapefiles with unicode encoded as UTF-16 and UTF-32, especially UTF-16-LE (if at all possible).* +When using other encodings, especially UTF-32 or UTF-16, a whole host of other known (and unknown) unicode related issues can be encountered. See ## Specialist Unicode Handling below for more details. -Before you can add records you must first create the fields that define what types of -values will go into each attribute. +#### Non-utf8 encodings +For reading shapefiles in any non-utf8 encoding, such as Latin-1, just +supply the encoding option when creating the Reader class. -There are several different field types, all of which support storing None values as NULL. -Text fields are created using the 'C' type, and the third 'size' argument can be customized to the expected -length of text values to save space: + >>> r = shapefile.Reader("tests/shapefiles/latin1.shp", encoding="latin1") + >>> r.record(0) == [2, u'Ñandú'] + True + +Once you have loaded the shapefile, you may choose to save it using another more supportive encoding such +as UTF-8. Assuming the new encoding supports the characters you are trying to write, reading it back in +should give you the same unicode string you started with. - >>> w = shapefile.Writer('tests/shapefiles/test/dtype') - >>> w.field('TEXT', 'C') - >>> w.field('SHORT_TEXT', 'C', size=5) - >>> w.field('LONG_TEXT', 'C', size=250) + >>> w = shapefile.Writer("tests/shapefiles/test/latin_as_utf8.shp", encoding="utf8") + >>> w.fields = r.fields[1:] + >>> w.record(*r.record(0)) >>> w.null() - >>> w.record('Hello', 'World', 'World'*50) + >>> r.close() >>> w.close() - >>> r = shapefile.Reader('tests/shapefiles/test/dtype') - >>> assert r.record(0) == ['Hello', 'World', 'World'*50] + >>> r = shapefile.Reader("tests/shapefiles/test/latin_as_utf8.shp", encoding="utf8") + >>> r.record(0) == [2, u'Ñandú'] + True >>> r.close() -Date fields are created using the 'D' type, and can be created using either -date objects, lists, or a YYYYMMDD formatted string. -Field length or decimal have no impact on this type: +If you supply the wrong encoding and the string is unable to be decoded, PyShp will by default raise an +exception. +#### Custom handling of encoding errors. +If however, on rare occasion, you are unable to find the correct encoding and want to ignore +or replace encoding errors, you can specify the "encodingErrors" to be used by the decode method. This +applies to both reading and writing. - >>> from datetime import date - >>> w = shapefile.Writer('tests/shapefiles/test/dtype') - >>> w.field('DATE', 'D') - >>> w.null() - >>> w.null() - >>> w.null() - >>> w.null() - >>> w.record(date(1898,1,30)) - >>> w.record([1998,1,30]) - >>> w.record('19980130') - >>> w.record(None) - >>> w.close() - >>> r = shapefile.Reader('tests/shapefiles/test/dtype') - >>> assert r.record(0) == [date(1898,1,30)] - >>> assert r.record(1) == [date(1998,1,30)] - >>> assert r.record(2) == [date(1998,1,30)] - >>> assert r.record(3) == [None] + >>> r = shapefile.Reader("tests/shapefiles/latin1.shp", encoding="ascii", encodingErrors="replace") + >>> r.record(0) == [2, u'�and�'] + True >>> r.close() -Numeric fields are created using the 'N' type (or the 'F' type, which is exactly the same). -By default the fourth decimal argument is set to zero, essentially creating an integer field. -To store floats you must set the decimal argument to the precision of your choice. -To store very large numbers you must increase the field length size to the total number of digits -(including comma and minus). - - - >>> w = shapefile.Writer('tests/shapefiles/test/dtype') - >>> w.field('INT', 'N') - >>> w.field('LOWPREC', 'N', decimal=2) - >>> w.field('MEDPREC', 'N', decimal=10) - >>> w.field('HIGHPREC', 'N', decimal=30) - >>> w.field('FTYPE', 'F', decimal=10) - >>> w.field('LARGENR', 'N', 101) - >>> nr = 1.3217328 - >>> w.null() - >>> w.null() - >>> w.record(INT=nr, LOWPREC=nr, MEDPREC=nr, HIGHPREC=-3.2302e-25, FTYPE=nr, LARGENR=int(nr)*10**100) - >>> w.record(None, None, None, None, None, None) - >>> w.close() +Valid values for encodingErrors are those supported by both `bytes.decode` and `str.encode` in the Python +version being used, e.g. 'strict', 'ignore' or 'replace' +in [CPython 3.9 - 3.14](https://docs.python.org/3/library/stdtypes.html#bytes.decode) +('xmlcharrefreplace' and 'backslashreplace' are only supported by +[`str.encode`](https://docs.python.org/3/library/stdtypes.html#str.encode)). - >>> r = shapefile.Reader('tests/shapefiles/test/dtype') - >>> assert r.record(0) == [1, 1.32, 1.3217328, -3.2302e-25, 1.3217328, 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] - >>> assert r.record(1) == [None, None, None, None, None, None] - >>> r.close() +## Specialist Unicode handling. +### Summary +If at all possible use UTF-8. Or at the very least avoid UTF-16 and UTF-32 +### Background +Unicode is one of humanity's greatest achievements. GIS applications support it within Shapefiles. +But most unicode encodings (e.g. UTF-8, UTF-16) assume +data will be stored in pure binary form, as bytes (8-bit octets). The Shapefile spec however, defers to +a long lost dbf spec (the URL link to a former company's website is now dead) which like most other dbf +specifications, will require text to be stored as Ascii. +The fields are some known width, and Extended Ascii character sets (with 256 entries, not only the first +128 common to them all) are essentially one-to-one mappings to bytes - so this would not be a +problem in isolation. +### Padding bytes +Dbf field names are post padded with null bytes, and dbf text fields ("C" and "M") are post +padded with Ascii spaces (the byte 0x20). Example shapefiles exist where text fields also end in numerous +null bytes, essentially also as padding. +#### Problem 1. +Encodings of some unicode strings may contain these exact padding bytes as part of their data (in particular +the first null byte is not necessarily a null terminator). + >>> "囊萤映雪".encode("utf-16-be") + b'V\xca\x84$f \x96\xea' -Finally, we can create boolean fields by setting the type to 'L'. -This field can take True or False values, or 1 (True) or 0 (False). -None is interpreted as missing. +#### Problem 2. +Some unicode strings encode to byte sequences under some non-UTF-8 codecs, that contain and even +trail a null byte as part of their data. + >>> "ABC".encode("utf-16-le") + b'A\x00B\x00C\x00' - >>> w = shapefile.Writer('tests/shapefiles/test/dtype') - >>> w.field('BOOLEAN', 'L') - >>> w.null() - >>> w.null() - >>> w.null() - >>> w.null() - >>> w.null() - >>> w.null() - >>> w.record(True) - >>> w.record(1) - >>> w.record(False) - >>> w.record(0) - >>> w.record(None) - >>> w.record("Nonsense") - >>> w.close() +#### Problem 3. +Some unicode strings encode to byte sequences under some non-UTF-8 codecs, that +are entirely dbf padding bytes, e.g.: - >>> r = shapefile.Reader('tests/shapefiles/test/dtype') - >>> r.record(0) - Record #0: [True] - >>> r.record(1) - Record #1: [True] - >>> r.record(2) - Record #2: [False] - >>> r.record(3) - Record #3: [False] - >>> r.record(4) - Record #4: [None] - >>> r.record(5) - Record #5: [None] - >>> r.close() + >>> '††††'.encode("utf-16-le") + b' ' -You can also add attributes using keyword arguments where the keys are field names. +#### Problem 4. +If encodings of even only moderately long unicode strings (especially the 10 byte field names) are truncated at byte +boundaries (i.e. not truncated correctly at one of the string's code-point boundaries), their data is corrupted, and +the encoding of the final code point whose bytes were chopped is corrupted. + >>> try: + ... "ÀÀÀÀ०".encode()[:10].decode() + ... except UnicodeDecodeError: + ... print("Failed to decode") + ... + Failed to decode - >>> w = shapefile.Writer('tests/shapefiles/test/dtype') - >>> w.field('FIRST_FLD','C', 40) - >>> w.field('SECOND_FLD','C', 40) - >>> w.null() - >>> w.null() - >>> w.record('First', 'Line') - >>> w.record(FIRST_FLD='First', SECOND_FLD='Line') - >>> w.close() +#### Problem 5. +Historically, PyShp has attempted to prettify dbf string data. E.g. stripping whitespace. +In particular, ascii spaces (`b" "`) in a dbf name were transformed to underscores (`b"_"`) - but +crucially, only after encoding to bytes, whether the codec was ascii or not. This can result in silent +changes to the user's data, and since version 3.1 PyShp no longer does this. -### Adding Geometry + >>> s = "囊萤映雪" + >>> s2 = s.encode("utf-16-be").replace(b" ",b"_").decode("utf-16-be") + >>> print(s2) + 囊萤晟雪 + >>> s == s2 + False -Geometry is added using one of several convenience methods. The "null" method is used -for null shapes, "point" is used for point shapes, "multipoint" is used for multipoint shapes, "line" for lines, -"poly" for polygons. +### PyShp's approach +The simplest, most robust solution to store arbitrary codec encodings as ascii bytes, is probably to encode those +encodings as Base64 (with an alphabet excluding dbf pad bytes), and then encode that Base64 string a third time as Ascii bytes. +PyShp tries to be a little easier to use than this. The intention is to adhere to [Postel's Law](https://en.wikipedia.org/wiki/Robustness_principle) so hopefully PyShp is still fairly lenient when decoding and reading Shapefiles. But either informative or +stricter, when the user attempts to encode or write potentially invalid text data. +#### UTF-8 and Single byte codecs +UTF-8 is backwards compatible with ascii, and so far (as we know), is only known to be affected by Problem 4. +PyShp a provides a faster 'happy' path for Writers and DbfWriters using "utf8" (the default), and when text +field values are known to have been encoded, each code point to a single byte. Otherwise, a more robust +strategy is used, to handle the trickier codecs, to hopefully correctly encode data as far as possible, +otherwise to raise a warning or error. +#### Warnings +When possible data corruption is detected when reading or writing a Shapefile (or dbf file), by default +PyShp will raise a warning. Supported situations are listed below under Decoding and Encoding. +#### Strict mode +If a Writer (or dbf version) is created with `strict=True` then an exception is raised instead of any of the above warnings +(hopefully before corrupt data can be written at all). +#### Decoding + - PyShp first removes all padding bytes. If decoding fails, those padding bytes are restored one by one to the data + until decoding succeeds. A warning is raised if any padding bytes were required, but an exception is not raised for this + in strict mode (those pad bytes might just have been part of a valid encoding). + - PyShp warns if there is a null character in a decoded string (but silently accepts null bytes in encoded strings). This + warning does not become an exception in strict mode (there might simply be null characters in the data). +#### Encoding + - PyShp warns when trying to write data to a text field or field name that PyShp itself would not be able to decode correctly. + - PyShp now truncates text fields the dbf field `size` or 10 byte limit for names, by if necessary, successively +truncating the actual string, one code point at a time, until the number of encoded bytes is less than or equal +to the maximum. If truncation was necessary, a warning is raised. + - When trying to write text data, that ends with code points that would encode + to pad bytes (these would be lost on decoding, otherwise all fields stored as utf-16 under the limit will end + in ††††††††††††...). + - PyShp warns if a field name is written containing a null character (before encoding). For users who wish to steer well clear of potential bugs. -**Adding a Null shape** +## Reading Large Shapefiles -A shapefile may contain some records for which geometry is not available, and may be set using the "null" method. -Because Null shape types (shape type 0) have no geometry the "null" method is called without any arguments. +Despite being a lightweight library, PyShp is designed to be able to read shapefiles of any size, allowing you to work with hundreds of thousands or even millions +of records and complex geometries. +### Iterating through a shapefile - >>> w = shapefile.Writer('tests/shapefiles/test/null') - >>> w.field('name', 'C') +As an example, let's load this Natural Earth shapefile of more than 4000 global administrative boundary polygons: - >>> w.null() - >>> w.record('nullgeom') - >>> w.close() + >>> sf = shapefile.Reader("https://archive.org/download/ne_10m_admin_1_states_provinces/ne_10m_admin_1_states_provinces.zip") -**Adding a Point shape** +When first creating the Reader class, the library only reads the header information +and leaves the rest of the file contents alone. Once you call the records() and shapes() +methods however, it will attempt to read the entire file into memory at once. +For very large files this can result in MemoryError. So when working with large files +it is recommended to use instead the iterShapes(), iterRecords(), or iterShapeRecords() +methods instead. These iterate through the file contents one at a time, enabling you to loop +through them while keeping memory usage at a minimum. -Point shapes are added using the "point" method. A point is specified by an x and -y value. + >>> for shape in sf.iterShapes(): + ... # do something here + ... pass - >>> w = shapefile.Writer('tests/shapefiles/test/point') - >>> w.field('name', 'C') + >>> for rec in sf.iterRecords(): + ... # do something here + ... pass - >>> w.point(122, 37) - >>> w.record('point1') + >>> for shapeRec in sf.iterShapeRecords(): + ... # do something here + ... pass - >>> w.close() + >>> for shapeRec in sf: # same as iterShapeRecords() + ... # do something here + ... pass -**Adding a MultiPoint shape** +### Limiting which fields to read -If your point data allows for the possibility of multiple points per feature, use "multipoint" instead. -These are specified as a list of xy point coordinates. +By default when reading the attribute records of a shapefile, pyshp unpacks and returns the data for all of the dbf fields, regardless of whether you actually need that data or not. To limit which field data is unpacked when reading each record and speed up processing time, you can specify the `fields` argument to any of the methods involving record data. Note that the order of the specified fields does not matter, the resulting records will list the specified field values in the order that they appear in the original dbf file. For instance, if we are only interested in the country and name of each admin unit, the following is a more efficient way of iterating through the file: - >>> w = shapefile.Writer('tests/shapefiles/test/multipoint') - >>> w.field('name', 'C') + >>> fields = ["geonunit", "name"] + >>> for rec in sf.iterRecords(fields=fields): + ... # do something + ... pass + >>> rec + Record #4595: ['Birgu', 'Malta'] - >>> w.multipoint([[122,37], [124,32]]) - >>> w.record('multipoint1') +### Attribute filtering - >>> w.close() +In many cases, we aren't interested in all entries of a shapefile, but rather only want to retrieve a small subset of records by filtering on some attribute. To avoid wasting time reading records and shapes that we don't need, we can start by iterating only the records and fields of interest, check if the record matches some condition as a way to filter the data, and finally load the full record and shape geometry for those that meet the condition: -**Adding a LineString shape** -For LineString shapefiles, each shape is given as a list of one or more linear features. -Each of the linear features must have at least two points. + >>> filter_field = "geonunit" + >>> filter_value = "Eritrea" + >>> for rec in sf.iterRecords(fields=[filter_field]): + ... if rec[filter_field] == filter_value: + ... # load full record and shape + ... shapeRec = sf.shapeRecord(rec.oid) + ... shapeRec.record["name"] + 'Debubawi Keyih Bahri' + 'Debub' + 'Semenawi Keyih Bahri' + 'Gash Barka' + 'Maekel' + 'Anseba' +Selectively reading only the necessary data in this way is particularly useful for efficiently processing a limited subset of data from very large files or when looping through a large number of files, especially if they contain large attribute tables or complex shape geometries. - >>> w = shapefile.Writer('tests/shapefiles/test/line') - >>> w.field('name', 'C') +### Spatial filtering - >>> w.line([ - ... [[1,5],[5,5],[5,1],[3,3],[1,1]], # line 1 - ... [[3,2],[2,6]] # line 2 - ... ]) +Another common use-case is that we only want to read those records that are located in some region of interest. Because the shapefile stores the bounding box of each shape separately from the geometry data, it's possible to quickly retrieve all shapes that might overlap a given bounding box region without having to load the full shape geometry data for every shape. This can be done by specifying the `bbox` argument to the shapes, iterShapes, or iterShapeRecords methods: - >>> w.record('linestring1') - >>> w.close() - -**Adding a Polygon shape** - -Similarly to LineString, Polygon shapes consist of multiple polygons, and must be given as a list of polygons. -The main difference is that polygons must have at least 4 points and the last point must be the same as the first. -It's also okay if you forget to repeat the first point at the end; PyShp automatically checks and closes the polygons -if you don't. - -It's important to note that for Polygon shapefiles, your polygon coordinates must be ordered in a clockwise direction. -If any of the polygons have holes, then the hole polygon coordinates must be ordered in a counterclockwise direction. -The direction of your polygons determines how shapefile readers will distinguish between polygon outlines and holes. + >>> bbox = [36.423, 12.360, 43.123, 18.004] # ca bbox of Eritrea + >>> fields = ["geonunit","name"] + >>> for shapeRec in sf.iterShapeRecords(bbox=bbox, fields=fields): + ... shapeRec.record + Record #368: ['Afar', 'Ethiopia'] + Record #369: ['Tadjourah', 'Djibouti'] + Record #375: ['Obock', 'Djibouti'] + Record #376: ['Debubawi Keyih Bahri', 'Eritrea'] + Record #1106: ['Amhara', 'Ethiopia'] + Record #1107: ['Gedarif', 'Sudan'] + Record #1108: ['Tigray', 'Ethiopia'] + Record #1414: ['Sa`dah', 'Yemen'] + Record #1415: ['`Asir', 'Saudi Arabia'] + Record #1416: ['Hajjah', 'Yemen'] + Record #1417: ['Jizan', 'Saudi Arabia'] + Record #1598: ['Debub', 'Eritrea'] + Record #1599: ['Red Sea', 'Sudan'] + Record #1600: ['Semenawi Keyih Bahri', 'Eritrea'] + Record #1601: ['Gash Barka', 'Eritrea'] + Record #1602: ['Kassala', 'Sudan'] + Record #1603: ['Maekel', 'Eritrea'] + Record #2037: ['Al Hudaydah', 'Yemen'] + Record #3741: ['Anseba', 'Eritrea'] +This functionality means that shapefiles can be used as a bare-bones spatially indexed database, with very fast bounding box queries for even the largest of shapefiles. Note that, as with all spatial indexing, this method does not guarantee that the *geometries* of the resulting matches overlap the queried region, only that their *bounding boxes* overlap. - >>> w = shapefile.Writer('tests/shapefiles/test/polygon') - >>> w.field('name', 'C') - >>> w.poly([ - ... [[113,24], [112,32], [117,36], [122,37], [118,20]], # poly 1 - ... [[116,29],[116,26],[119,29],[119,32]], # hole 1 - ... [[15,2], [17,6], [22,7]] # poly 2 - ... ]) - >>> w.record('polygon1') - >>> w.close() +## Writing large shapefiles -**Adding from an existing Shape object** +Similar to the Reader class, the shapefile Writer class uses a streaming approach to keep memory +usage at a minimum and allow writing shapefiles of arbitrarily large sizes. The library takes care of this under-the-hood by immediately +writing each geometry and record to disk the moment they +are added using shape() or record(). Once the writer is closed, exited, or garbage +collected, the final header information is calculated and written to the beginning of +the file. -Finally, geometry can be added by passing an existing "Shape" object to the "shape" method. -You can also pass it any GeoJSON dictionary or _\_geo_interface\_\_ compatible object. -This can be particularly useful for copying from one file to another: +### Merging multiple shapefiles +This means that it's possible to merge hundreds or thousands of shapefiles, as +long as you iterate through the source files to avoid loading everything into +memory. The following example copies the contents of a shapefile to a new file 10 times: - >>> r = shapefile.Reader('tests/shapefiles/test/polygon') + >>> # create writer + >>> w = shapefile.Writer('tests/shapefiles/test/merge') - >>> w = shapefile.Writer('tests/shapefiles/test/copy') - >>> w.fields = r.fields[1:] # skip first deletion field + >>> # copy over fields from the reader + >>> r = shapefile.Reader("tests/shapefiles/blockgroups") + >>> for field in r.fields[1:]: + ... w.field(*field) - >>> # adding existing Shape objects - >>> for shaperec in r.iterShapeRecords(): - ... w.record(*shaperec.record) - ... w.shape(shaperec.shape) + >>> # copy the shapefile to writer 10 times + >>> repeat = 10 + >>> for i in range(repeat): + ... r = shapefile.Reader("tests/shapefiles/blockgroups") + ... for shapeRec in r.iterShapeRecords(): + ... w.record(*shapeRec.record) + ... w.shape(shapeRec.shape) - >>> # or GeoJSON dicts - >>> for shaperec in r.iterShapeRecords(): - ... w.record(*shaperec.record) - ... w.shape(shaperec.shape.__geo_interface__) + >>> # check that the written file is 10 times longer + >>> len(w) == len(r) * 10 + True - >>> r.close() + >>> # close the writer >>> w.close() +In this trivial example, we knew that all files had the exact same field names, ordering, and types. In other scenarios, you will have to additionally make sure that all shapefiles have the exact same fields in the same order, and that they all contain the same geometry type. -### Geometry and Record Balancing +### Editing shapefiles -Because every shape must have a corresponding record it is critical that the -number of records equals the number of shapes to create a valid shapefile. You -must take care to add records and shapes in the same order so that the record -data lines up with the geometry data. For example: +If you need to edit a shapefile you would have to read the +file one record at a time, modify or filter the contents, and write it back out. For instance, to create a copy of a shapefile that only keeps a subset of relevant fields: + >>> # create writer + >>> w = shapefile.Writer('tests/shapefiles/test/edit') - >>> w = shapefile.Writer('tests/shapefiles/test/balancing', shapeType=shapefile.POINT) - >>> w.field("field1", "C") - >>> w.field("field2", "C") + >>> # define which fields to keep + >>> keep_fields = ['BKG_KEY', 'MEDIANRENT'] - >>> w.record("row", "one") - >>> w.point(1, 1) + >>> # copy over the relevant fields from the reader + >>> r = shapefile.Reader("tests/shapefiles/blockgroups") + >>> for field in r.fields[1:]: + ... if field[0] in keep_fields: + ... w.field(*field) - >>> w.record("row", "two") - >>> w.point(2, 2) + >>> # write only the relevant attribute values + >>> for shapeRec in r.iterShapeRecords(fields=keep_fields): + ... w.record(*shapeRec.record) + ... w.shape(shapeRec.shape) -To help prevent accidental misalignment PyShp has an "auto balance" feature to -make sure when you add either a shape or a record the two sides of the -equation line up. This way if you forget to update an entry the -shapefile will still be valid and handled correctly by most shapefile -software. Autobalancing is NOT turned on by default. To activate it set -the attribute autoBalance to 1 or True: + >>> # close writer + >>> w.close() +## 3D and Other Geometry Types - >>> w.autoBalance = 1 - >>> w.record("row", "three") - >>> w.record("row", "four") - >>> w.point(4, 4) +Most shapefiles store conventional 2D points, lines, or polygons. But the shapefile format is also capable +of storing various other types of geometries as well, including complex 3D surfaces and objects. - >>> w.recNum == w.shpNum - True +### Shapefiles with measurement (M) values -You also have the option of manually calling the balance() method at any time -to ensure the other side is up to date. When balancing is used -null shapes are created on the geometry side or records -with a value of "NULL" for each field is created on the attribute side. -This gives you flexibility in how you build the shapefile. -You can create all of the shapes and then create all of the records or vice versa. +Measured shape types are shapes that include a measurement value at each vertex, for instance +speed measurements from a GPS device. Shapes with measurement (M) values are added with the following +methods: "pointm", "multipointm", "linem", and "polygonm". The M-values are specified by adding a +third M value to each XY coordinate. Missing or unobserved M-values are specified with a None value, +or by simply omitting the third M-coordinate. - >>> w.autoBalance = 0 - >>> w.record("row", "five") - >>> w.record("row", "six") - >>> w.record("row", "seven") - >>> w.point(5, 5) - >>> w.point(6, 6) - >>> w.balance() + >>> w = shapefile.Writer('tests/shapefiles/test/linem') + >>> w.field('name', 'C') - >>> w.recNum == w.shpNum - True + >>> w.linem([ + ... [[1,5,0],[5,5],[5,1,3],[3,3,None],[1,1,0]], # line with one omitted and one missing M-value + ... [[3,2],[2,6]] # line without any M-values + ... ]) -If you do not use the autoBalance() or balance() method and forget to manually -balance the geometry and attributes the shapefile will be viewed as corrupt by -most shapefile software. + >>> w.record('linem1') -### Writing .prj files -A .prj file, or projection file, is a simple text file that stores a shapefile's map projection and coordinate reference system to help mapping software properly locate the geometry on a map. If you don't have one, you may get confusing errors when you try and use the shapefile you created. The GIS software may complain that it doesn't know the shapefile's projection and refuse to accept it, it may assume the shapefile is the same projection as the rest of your GIS project and put it in the wrong place, or it might assume the coordinates are an offset in meters from latitude and longitude 0,0 which will put your data in the middle of the ocean near Africa. The text in the .prj file is a [Well-Known-Text (WKT) projection string](https://en.wikipedia.org/wiki/Well-known_text_representation_of_coordinate_reference_systems). Projection strings can be quite long so they are often referenced using numeric codes call EPSG codes. The .prj file must have the same base name as your shapefile. So for example if you have a shapefile named "myPoints.shp", the .prj file must be named "myPoints.prj". + >>> w.close() -If you're using the same projection over and over, the following is a simple way to create the .prj file assuming your base filename is stored in a variable called "filename": +Shapefiles containing M-values can be examined in several ways: -``` - with open("{}.prj".format(filename), "w") as prj: - wkt = 'GEOGCS["WGS 84",' - wkt += 'DATUM["WGS_1984",' - wkt += 'SPHEROID["WGS 84",6378137,298.257223563]]' - wkt += ',PRIMEM["Greenwich",0],' - wkt += 'UNIT["degree",0.0174532925199433]]' - prj.write(wkt) -``` + >>> r = shapefile.Reader('tests/shapefiles/test/linem') -If you need to dynamically fetch WKT projection strings, you can use the pure Python [PyCRS](https://github.com/karimbahgat/PyCRS) module which has a number of useful features. + >>> r.mbox # the lower and upper bound of M-values in the shapefile + MBox(mmin=0.0, mmax=3.0) -# Advanced Use + >>> r.shape(0).m # flat list of M-values + [0.0, None, 3.0, None, 0.0, None, None] -## Common Errors and Fixes + >>> r.close() -Below we list some commonly encountered errors and ways to fix them. -### Warnings and Logging +### Shapefiles with elevation (Z) values -By default, PyShp chooses to be transparent and provide the user with logging information and warnings about non-critical issues when reading or writing shapefiles. This behavior is controlled by the module constant `VERBOSE` (which defaults to True). If you would rather suppress this information, you can simply set this to False: +Elevation shape types are shapes that include an elevation value at each vertex, for instance elevation from a GPS device. +Shapes with elevation (Z) values are added with the following methods: "pointz", "multipointz", "linez", and "polyz". +The Z-values are specified by adding a third Z value to each XY coordinate. Z-values do not support the concept of missing data, +but if you omit the third Z-coordinate it will default to 0. Note that Z-type shapes also support measurement (M) values added +as a fourth M-coordinate. This too is optional. - >>> shapefile.VERBOSE = False + >>> w = shapefile.Writer('tests/shapefiles/test/linez') + >>> w.field('name', 'C') -All logging happens under the namespace `shapefile`. So another way to suppress all PyShp warnings would be to alter the logging behavior for that namespace: + >>> w.linez([ + ... [[1,5,18],[5,5,20],[5,1,22],[3,3],[1,1]], # line with some omitted Z-values + ... [[3,2],[2,6]], # line without any Z-values + ... [[3,2,15,0],[2,6,13,3],[1,9,14,2]] # line with both Z- and M-values + ... ]) + >>> w.record('linez1') - >>> import logging - >>> logging.getLogger('shapefile').setLevel(logging.ERROR) + >>> w.close() -### Shapefile Encoding Errors +To examine a Z-type shapefile you can do: -PyShp supports reading and writing shapefiles in any language or character encoding, and provides several options for decoding and encoding text. -Most shapefiles are written in UTF-8 encoding, PyShp's default encoding, so in most cases you don't have to specify the encoding. -#### Encoding errors -If you encounter an encoding error when reading a shapefile, this means the shapefile was likely written in a non-utf8 encoding. -For instance, when working with English language shapefiles, a common reason for encoding errors is that the shapefile was written in Latin-1 encoding. -#### Recommendation -*We recommend avoiding writing Shapefiles with unicode encoded as UTF-16 and UTF-32, especially UTF-16-LE (if at all possible).* -When using other encodings, especially UTF-32 or UTF-16, a whole host of other known (and unknown) unicode related issues can be encountered. See ## Specialist Unicode Handling below for more details. + >>> r = shapefile.Reader('tests/shapefiles/test/linez') -#### Non-utf8 encodings -For reading shapefiles in any non-utf8 encoding, such as Latin-1, just -supply the encoding option when creating the Reader class. + >>> r.zbox # the lower and upper bound of Z-values in the shapefile + ZBox(zmin=0.0, zmax=22.0) + >>> r.shape(0).z # flat list of Z-values + [18.0, 20.0, 22.0, 0.0, 0.0, 0.0, 0.0, 15.0, 13.0, 14.0] - >>> r = shapefile.Reader("tests/shapefiles/latin1.shp", encoding="latin1") - >>> r.record(0) == [2, u'Ñandú'] - True + >>> r.shape(0).points_3D # list of 3D coordinates incorporating the Z-values + [(1.0, 5.0, 18.0), (5.0, 5.0, 20.0), (5.0, 1.0, 22.0), (3.0, 3.0, 0.0), (1.0, 1.0, 0.0), (3.0, 2.0, 0.0), (2.0, 6.0, 0.0), (3.0, 2.0, 15.0), (2.0, 6.0, 13.0), (1.0, 9.0, 14.0)] -Once you have loaded the shapefile, you may choose to save it using another more supportive encoding such -as UTF-8. Assuming the new encoding supports the characters you are trying to write, reading it back in -should give you the same unicode string you started with. + >>> r.close() +### 3D MultiPatch Shapefiles - >>> w = shapefile.Writer("tests/shapefiles/test/latin_as_utf8.shp", encoding="utf8") - >>> w.fields = r.fields[1:] - >>> w.record(*r.record(0)) - >>> w.null() - >>> r.close() - >>> w.close() +Multipatch shapes are useful for storing composite 3-Dimensional objects. +A MultiPatch shape represents a 3D object made up of one or more surface parts. +Each surface in "parts" is defined by a list of XYZM values (Z and M values optional), and its corresponding type is +given in the "partTypes" argument. The part type decides how the coordinate sequence is to be interpreted, and can be one +of the following module constants: TRIANGLE_STRIP, TRIANGLE_FAN, OUTER_RING, INNER_RING, FIRST_RING, or RING. +For instance, a TRIANGLE_STRIP may be used to represent the walls of a building, combined with a TRIANGLE_FAN to represent +its roof: - >>> r = shapefile.Reader("tests/shapefiles/test/latin_as_utf8.shp", encoding="utf8") - >>> r.record(0) == [2, u'Ñandú'] - True - >>> r.close() + >>> from shapefile import TRIANGLE_STRIP, TRIANGLE_FAN -If you supply the wrong encoding and the string is unable to be decoded, PyShp will by default raise an -exception. + >>> w = shapefile.Writer('tests/shapefiles/test/multipatch') + >>> w.field('name', 'C') -#### Custom handling of encoding errors. -If however, on rare occasion, you are unable to find the correct encoding and want to ignore -or replace encoding errors, you can specify the "encodingErrors" to be used by the decode method. This -applies to both reading and writing. + >>> w.multipatch([ + ... [[0,0,0],[0,0,3],[5,0,0],[5,0,3],[5,5,0],[5,5,3],[0,5,0],[0,5,3],[0,0,0],[0,0,3]], # TRIANGLE_STRIP for house walls + ... [[2.5,2.5,5],[0,0,3],[5,0,3],[5,5,3],[0,5,3],[0,0,3]], # TRIANGLE_FAN for pointed house roof + ... ], + ... partTypes=[TRIANGLE_STRIP, TRIANGLE_FAN]) # one type for each part + >>> w.record('house1') - >>> r = shapefile.Reader("tests/shapefiles/latin1.shp", encoding="ascii", encodingErrors="replace") - >>> r.record(0) == [2, u'�and�'] - True - >>> r.close() + >>> w.close() -Valid values for encodingErrors are those supported by both `bytes.decode` and `str.encode` in the Python -version being used, e.g. 'strict', 'ignore' or 'replace' -in [CPython 3.9 - 3.14](https://docs.python.org/3/library/stdtypes.html#bytes.decode) -('xmlcharrefreplace' and 'backslashreplace' are only supported by -[`str.encode`](https://docs.python.org/3/library/stdtypes.html#str.encode)). +For an introduction to the various multipatch part types and examples of how to create 3D MultiPatch objects see [this +ESRI White Paper](http://downloads.esri.com/support/whitepapers/ao_/J9749_MultiPatch_Geometry_Type.pdf). -## Specialist Unicode handling. -### Summary -If at all possible use UTF-8. Or at the very least avoid UTF-16 and UTF-32 -### Background -Unicode is one of humanity's greatest achievements. GIS applications support it within Shapefiles. -But most unicode encodings (e.g. UTF-8, UTF-16) assume -data will be stored in pure binary form, as bytes (8-bit octets). The Shapefile spec however, defers to -a long lost dbf spec (the URL link to a former company's website is now dead) which like most other dbf -specifications, will require text to be stored as Ascii. -The fields are some known width, and Extended Ascii character sets (with 256 entries, not only the first -128 common to them all) are essentially one-to-one mappings to bytes - so this would not be a -problem in isolation. -### Padding bytes -Dbf field names are post padded with null bytes, and dbf text fields ("C" and "M") are post -padded with Ascii spaces (the byte 0x20). Example shapefiles exist where text fields also end in numerous -null bytes, essentially also as padding. -#### Problem 1. -Encodings of some unicode strings may contain these exact padding bytes as part of their data (in particular -the first null byte is not necessarily a null terminator). - >>> "囊萤映雪".encode("utf-16-be") - b'V\xca\x84$f \x96\xea' -#### Problem 2. -Some unicode strings encode to byte sequences under some non-UTF-8 codecs, that contain and even -trail a null byte as part of their data. +# Testing - >>> "ABC".encode("utf-16-le") - b'A\x00B\x00C\x00' +The testing framework is pytest, and the tests are located in ./tests. +This includes an extensive set of unit tests of the various pyshp features in ./tests/test_shapefile.py, +and tests against various input data. +In the same folder as README.md and pyproject.toml, from the command line run: -#### Problem 3. -Some unicode strings encode to byte sequences under some non-UTF-8 codecs, that -are entirely dbf padding bytes, e.g.: +```shell +python -m pytest +``` - >>> '††††'.encode("utf-16-le") - b' ' +Additionally, all the code and examples located in this file, README.md, +is tested and verified with the builtin doctest framework. +A special doctest runner will run the doctests, with test artefacts confined to a +a temporary directory, and optionally switching the remote URLs for localhost: -#### Problem 4. -If encodings of even only moderately long unicode strings (especially the 10 byte field names) are truncated at byte -boundaries (i.e. not truncated correctly at one of the string's code-point boundaries), their data is corrupted, and -the encoding of the final code point whose bytes were chopped is corrupted. +```shell +python tests/run_doctests.py +``` - >>> try: - ... "ÀÀÀÀ०".encode()[:10].decode() - ... except UnicodeDecodeError: - ... print("Failed to decode") - ... - Failed to decode -#### Problem 5. -Historically, PyShp has attempted to prettify dbf string data. E.g. stripping whitespace. -In particular, ascii spaces (`b" "`) in a dbf name were transformed to underscores (`b"_"`) - but -crucially, only after encoding to bytes, whether the codec was ascii or not. This can result in silent -changes to the user's data, and since version 3.1 PyShp no longer does this. +## Network tests - >>> s = "囊萤映雪" - >>> s2 = s.encode("utf-16-be").replace(b" ",b"_").decode("utf-16-be") - >>> print(s2) - 囊萤晟雪 - >>> s == s2 - False +Some of the tests and doctests, are intended to test reading shapefiles from +remote servers, which requires internet connectivity. The pytest tests are marked "network". +For rapid iteration, in CI, or when developing in offline testing environments, these +tests can be dealt with in two ways: + i) by skipping the network tests via : +```shell +pytest -m "not network" +``` +or the doctests via: +```shell +python tests/run_doctests.py -m "not network" +``` +or ii) by cloning a repo of the files they download, serving these on localhost in a separate process, +and running the network tests with the environment variable REPLACE_REMOTE_URLS_WITH_LOCALHOST to `yes`: +Setup a local file server (*): +``` +git clone http://github.com/JamesParrott/PyShp_test_shapefile +cd PyShp_test_shapefile +python -m http.server 8000 +``` +and then: +```bash +REPLACE_REMOTE_URLS_WITH_LOCALHOST=yes && pytest +``` +or the doctests via: +```bash +REPLACE_REMOTE_URLS_WITH_LOCALHOST=yes && python tests/run_doctests.py +``` +The network tests alone can also be run (without also running all the tests that don't +make network requests) using: `pytest -m network` (or the doctests using: `python tests/run_doctests.py -m network`). -### PyShp's approach -The simplest, most robust solution to store arbitrary codec encodings as ascii bytes, is probably to encode those -encodings as Base64 (with an alphabet excluding dbf pad bytes), and then encode that Base64 string a third time as Ascii bytes. -PyShp tries to be a little easier to use than this. The intention is to adhere to [Postel's Law](https://en.wikipedia.org/wiki/Robustness_principle) so hopefully PyShp is still fairly lenient when decoding and reading Shapefiles. But either informative or -stricter, when the user attempts to encode or write potentially invalid text data. -#### UTF-8 and Single byte codecs -UTF-8 is backwards compatible with ascii, and so far (as we know), is only known to be affected by Problem 4. -PyShp a provides a faster 'happy' path for Writers and DbfWriters using "utf8" (the default), and when text -field values are known to have been encoded, each code point to a single byte. Otherwise, a more robust -strategy is used, to handle the trickier codecs, to hopefully correctly encode data as far as possible, -otherwise to raise a warning or error. -#### Warnings -When possible data corruption is detected when reading or writing a Shapefile (or dbf file), by default -PyShp will raise a warning. Supported situations are listed below under Decoding and Encoding. -#### Strict mode -If a Writer (or dbf version) is created with `strict=True` then an exception is raised instead of any of the above warnings -(hopefully before corrupt data can be written at all). -#### Decoding - - PyShp first removes all padding bytes. If decoding fails, those padding bytes are restored one by one to the data - until decoding succeeds. A warning is raised if any padding bytes were required, but an exception is not raised for this - in strict mode (those pad bytes might just have been part of a valid encoding). - - PyShp warns if there is a null character in a decoded string (but silently accepts null bytes in encoded strings). This - warning does not become an exception in strict mode (there might simply be null characters in the data). -#### Encoding - - PyShp warns when trying to write data to a text field or field name that PyShp itself would not be able to decode correctly. - - PyShp now truncates text fields the dbf field `size` or 10 byte limit for names, by if necessary, successively -truncating the actual string, one code point at a time, until the number of encoded bytes is less than or equal -to the maximum. If truncation was necessary, a warning is raised. - - When trying to write text data, that ends with code points that would encode - to pad bytes (these would be lost on decoding, otherwise all fields stored as utf-16 under the limit will end - in ††††††††††††...). - - PyShp warns if a field name is written containing a null character (before encoding). For users who wish to steer well clear of potential bugs. -## Reading Large Shapefiles -Despite being a lightweight library, PyShp is designed to be able to read shapefiles of any size, allowing you to work with hundreds of thousands or even millions -of records and complex geometries. +# Contributors -### Iterating through a shapefile +``` +Atle Frenvik Sveen +Bas Couwenberg +Ben Beasley +Casey Meisenzahl +Charles Arnold +David A. Riggs +davidh-ssec +Edward Kawas +Evan Heidtmann +ezcitron +Felix Oesterle +fiveham +geospatialpython +Hannes +Ignacio Martinez Vazquez +James Parrott +Jason Moujaes +Jonty Wareing +Karim Bahgat +karanrn +Kurt Schwehr +Kyle Kelley +Lionel Guez +Louis Tiao +Marcin Cuprjak +mcuprjak +Micah Cochran +Michael Davis +Michal Čihař +Mike Toews +Miroslav Šedivý +Nilo +pakoun +Paula Spannring +Paulo Ernesto +Raynor Vliegendhart +Razzi Abuissa +RosBer97 +Ross Rogers +Ryan Brideau +Thomas Beierlein +Tim Gates +Tobias Megies +Tommi Penttinen +Uli Köhler +Vsevolod Novikov +Zac Miller +``` -As an example, let's load this Natural Earth shapefile of more than 4000 global administrative boundary polygons: +# Older version changes +## 3.1.4 +### Bug fix + - Fix bug causing dates supplied as length 8 strings of digits to be encoded by the custom encoding, not ascii. - >>> sf = shapefile.Reader("https://archive.org/download/ne_10m_admin_1_states_provinces/ne_10m_admin_1_states_provinces.zip") +### Testing + - Test other codecs (ascii and UTF-8, UTF-16 & UTF-32 so far). + - Test all available codecs in CI (92 of them). -When first creating the Reader class, the library only reads the header information -and leaves the rest of the file contents alone. Once you call the records() and shapes() -methods however, it will attempt to read the entire file into memory at once. -For very large files this can result in MemoryError. So when working with large files -it is recommended to use instead the iterShapes(), iterRecords(), or iterShapeRecords() -methods instead. These iterate through the file contents one at a time, enabling you to loop -through them while keeping memory usage at a minimum. +## 3.1.3 + - Restore faster text writing paths for single-byte Ascii encodings, and Utf-8. + +## 3.1.2 + - Raise error in strict mode when creating field with a name, or writing strings, that ends with whole code points + that whose encoding is pad bytes. + +## 3.1.1 +### Unicode support made even more robust and yet another encoding bug fixed! + - When reading, only use minimum number of pad bytes to decode text successfully (fixes issue 423). + - When writing, warn (or raise in strict mode) if the text's encoding ends in pad bytes. + +## 3.1.0 +### Unicode support made more robust and encoding bugs fixed + - Truncation of field names and text fields now respects unicode code point boundaries (fixes issues - + 416 and 148). + - Warnings raised if truncation, or replacing b" " with b"_" would corrupt encoded field names, - + both if they would either be undecodable, or would silently decode to incorrect data (warns users + if issue 421 applies). + - Correctly truncated field names are now stored in field instances, as would actually be seen in the file. + - New strict mode. Writer(strict=True) raises errors or refuse to create fields and text records with data that - + would be truncated or cannot be correctly decoded back again by PyShp, exactly as given by the user. + - In strict mode, ascii spaces in encoded names are no longer replaced by ascii underscores at all + (work around to avoid corrupting unicode field names - provides opt-in fix for issue 421). + - BREAKING. When reading .dbf files, Trailing ascii spaces in text fields before a null terminator char (in the - + decoded string) is now removed (i.e. instead of .strip().rstrip('\x00') we now do: .rstrip("\x00").rstrip(" ")). + - BREAKING. Enclosing Whitespace other than trailing ascii spaces (0x20) after null chars in text fields is now - + preserved, when reading .dbf files (fixes issue 418 - James feels this was a bug. Let him know if you think otherwise). + - BREAKING Trailing null chars other than null terminators & null padding bytes, followed by whitespace other than + ascii spaces, are now preserved. + - Writing dbf records is now atomic. + +### ShpWriter.shape API Tweak (small breaking change). + - Make ShpWriter.shape return shape length in bytes (the + same as for offset) not in 16 bit words. +### Testing + - Include NullShapes in shp round trip test. + - En/decoding of Dbf files and Fields round trips correctly. + + +## 3.0.13 +### Bug fix + - Fix bug when reading empty shp files. +### Testing + - Add round trip tests for Multipatch and shp files (both passed). +## 3.0.12 +### Data consistency + - Add Shape.points_2D and Shape.points_3D properties - lists of guaranteed length tuples (2 and 3 respectively). - >>> for shape in sf.iterShapes(): - ... # do something here - ... pass +### Testing + - Round trip property-based tests for Polylines and Polygons (both pass). - >>> for rec in sf.iterRecords(): - ... # do something here - ... pass +## 3.0.11 +### Edge case handling + - Raise ShapefileException i) when creating Non-null Shapes without (or with empty) points + and ii) when creating Null Shapes with non-empty points. + - Ensure Shape.z and Shape.partTypes are _Arrays. + - Make Shape stricter about its args, e.g. only points or lines, only one point for Points. - >>> for shapeRec in sf.iterShapeRecords(): - ... # do something here - ... pass +### Bug fixes + - Multipoints with only a single point, now have their bbox calculated. - >>> for shapeRec in sf: # same as iterShapeRecords() - ... # do something here - ... pass +### Testing + - Round trip property-based tests for Multipoints (passes). -### Limiting which fields to read +## 3.0.10 +### Bug fix + - Convert directly supplied m values to None if they are strictly below ISDATA_LOWER_BOUND (-1e38). +### Testing + - Move tests into ./tests. + - Remove doctest runner from user land. + - Add round trip property tests for Point and PointM using hypothesis. -By default when reading the attribute records of a shapefile, pyshp unpacks and returns the data for all of the dbf fields, regardless of whether you actually need that data or not. To limit which field data is unpacked when reading each record and speed up processing time, you can specify the `fields` argument to any of the methods involving record data. Note that the order of the specified fields does not matter, the resulting records will list the specified field values in the order that they appear in the original dbf file. For instance, if we are only interested in the country and name of each admin unit, the following is a more efficient way of iterating through the file: +## 3.0.9 +### Testing + - Try to make tests not rely on downloads from Github repo URLs, to avoid 404s & 426s due to rate limits. +### Testability / separation of concerns: + - Separate shx and shp only methods into new ShpWriter, ShxWriter, ShpReader, and ShxReader classes +(instances of each pair is owned by the regular Shapefile Writer and Reader classes respectively). - >>> fields = ["geonunit", "name"] - >>> for rec in sf.iterRecords(fields=fields): - ... # do something - ... pass - >>> rec - Record #4595: ['Birgu', 'Malta'] -### Attribute filtering +### Bug fixes + - Remove ambiguity in API (BREAKING CHANGE). Reader and Writer now both raise TypeError + when passed a supported Shapefile target + as the positional-only, non-keyword arg, but also passed kwargs for shp, dbf or shx (instead + of silently dropping or overwriting those kwargs, with the user being none the wiser). + - Bug fix (BREAKING CHANGE). Reader and Writer now raise TypeError instead of continuing silently + as if given no args (e.g. in anticipation of a subsequent call to Reader.load), when passed an unsupported + truthy Python object (not a str or a Path) as its first arg (the positional-only, non-keyword arg). -In many cases, we aren't interested in all entries of a shapefile, but rather only want to retrieve a small subset of records by filtering on some attribute. To avoid wasting time reading records and shapes that we don't need, we can start by iterating only the records and fields of interest, check if the record matches some condition as a way to filter the data, and finally load the full record and shape geometry for those that meet the condition: + - BREAKING CHANGE. Correct NODATA lower threshold to -1e38 (as per [spec](https://www.esri.com/content/dam/esrisites/sitecore-archive/Files/Pdfs/library/whitepapers/pdfs/shapefile.pdf) 2, Numeric Types). This is 10% of + PyShp's previous threshold -10e38, so it is possible large negative values x (-1e39 < x < -1e38 will now be NODATA. - >>> filter_field = "geonunit" - >>> filter_value = "Eritrea" - >>> for rec in sf.iterRecords(fields=[filter_field]): - ... if rec[filter_field] == filter_value: - ... # load full record and shape - ... shapeRec = sf.shapeRecord(rec.oid) - ... shapeRec.record["name"] - 'Debubawi Keyih Bahri' - 'Debub' - 'Semenawi Keyih Bahri' - 'Gash Barka' - 'Maekel' - 'Anseba' +### Security + - Add python_version >= 3.10 to pre-commit dev & lint groups' 'dependency' (due to CVE in filelock 3.19). -Selectively reading only the necessary data in this way is particularly useful for efficiently processing a limited subset of data from very large files or when looping through a large number of files, especially if they contain large attribute tables or complex shape geometries. +## 3.0.8 -### Spatial filtering +### Testability / separation of concerns. + - Separate dbf only writing methods into a new dbfWriter class (an instance of which is owned by the regular Shapefile Writer class). -Another common use-case is that we only want to read those records that are located in some region of interest. Because the shapefile stores the bounding box of each shape separately from the geometry data, it's possible to quickly retrieve all shapes that might overlap a given bounding box region without having to load the full shape geometry data for every shape. This can be done by specifying the `bbox` argument to the shapes, iterShapes, or iterShapeRecords methods: +## 3.0.7 +### Testability / separation of concerns. + - Separate dbf only reading methods into a new dbfReader class (an instance of which is owned by the regular Shapefile Reader class). - >>> bbox = [36.423, 12.360, 43.123, 18.004] # ca bbox of Eritrea - >>> fields = ["geonunit","name"] - >>> for shapeRec in sf.iterShapeRecords(bbox=bbox, fields=fields): - ... shapeRec.record - Record #368: ['Afar', 'Ethiopia'] - Record #369: ['Tadjourah', 'Djibouti'] - Record #375: ['Obock', 'Djibouti'] - Record #376: ['Debubawi Keyih Bahri', 'Eritrea'] - Record #1106: ['Amhara', 'Ethiopia'] - Record #1107: ['Gedarif', 'Sudan'] - Record #1108: ['Tigray', 'Ethiopia'] - Record #1414: ['Sa`dah', 'Yemen'] - Record #1415: ['`Asir', 'Saudi Arabia'] - Record #1416: ['Hajjah', 'Yemen'] - Record #1417: ['Jizan', 'Saudi Arabia'] - Record #1598: ['Debub', 'Eritrea'] - Record #1599: ['Red Sea', 'Sudan'] - Record #1600: ['Semenawi Keyih Bahri', 'Eritrea'] - Record #1601: ['Gash Barka', 'Eritrea'] - Record #1602: ['Kassala', 'Sudan'] - Record #1603: ['Maekel', 'Eritrea'] - Record #2037: ['Al Hudaydah', 'Yemen'] - Record #3741: ['Anseba', 'Eritrea'] -This functionality means that shapefiles can be used as a bare-bones spatially indexed database, with very fast bounding box queries for even the largest of shapefiles. Note that, as with all spatial indexing, this method does not guarantee that the *geometries* of the resulting matches overlap the queried region, only that their *bounding boxes* overlap. +## 3.0.6 +### URL Downloading + - Unify tempfile creation and shapefile download logic. + - Check "Content-Type" header and sniff initial bytes + of response in order to possibly reject html responses, before parsing as a shapefile, to give a more useful error to users. + - Special case shapefiles hosted in Github repos to suggest appending the query string `?raw=true`. +### Testing: + - Add shapefile from Open Natual Hazard Modelling ([Paula Spannring](https://github.com/PaulaSp3) + and [Felix Oesterle, Austrian Research Centre for Forests](https://orcid.org/0000-0002-7772-6884)). -## Writing large shapefiles +## 3.0.5 -Similar to the Reader class, the shapefile Writer class uses a streaming approach to keep memory -usage at a minimum and allow writing shapefiles of arbitrarily large sizes. The library takes care of this under-the-hood by immediately -writing each geometry and record to disk the moment they -are added using shape() or record(). Once the writer is closed, exited, or garbage -collected, the final header information is calculated and written to the beginning of -the file. +### Project structure: + - Use dependency groups for dev and test instead of optional-dependencies in pyproject.toml ([Mike Toews](https://github.com/mwtoews)) -### Merging multiple shapefiles -This means that it's possible to merge hundreds or thousands of shapefiles, as -long as you iterate through the source files to avoid loading everything into -memory. The following example copies the contents of a shapefile to a new file 10 times: +## 3.0.4 - >>> # create writer - >>> w = shapefile.Writer('tests/shapefiles/test/merge') +### Type checking + - Fix pyshp-stubs, and add test to it. Remove all actual stubs from it, and replace with empty "shapefile-stubs/py.typed". Closes Issue 382 (https://github.com/GeospatialPython/pyshp/issues/382) and discussion 371 (https://github.com/GeospatialPython/pyshp/discussions/371). Bug originally found by Thomas Beierlein (https://github.com/GeospatialPython/pyshp/issues/367) - >>> # copy over fields from the reader - >>> r = shapefile.Reader("tests/shapefiles/blockgroups") - >>> for field in r.fields[1:]: - ... w.field(*field) - >>> # copy the shapefile to writer 10 times - >>> repeat = 10 - >>> for i in range(repeat): - ... r = shapefile.Reader("tests/shapefiles/blockgroups") - ... for shapeRec in r.iterShapeRecords(): - ... w.record(*shapeRec.record) - ... w.shape(shapeRec.shape) +## 3.0.3 - >>> # check that the written file is 10 times longer - >>> len(w) == len(r) * 10 - True +### Type checking +- Add optional dependency, "stubs", containing the package "pyshp-stubs" generated with stubgen (to avoid +including py.typed, and no longer being a single file project). - >>> # close the writer - >>> w.close() +### Bug fix +- Prevented UnboundLocalError when reading non-single point M and Z type Shapefiles (@ekawas-vrify). -In this trivial example, we knew that all files had the exact same field names, ordering, and types. In other scenarios, you will have to additionally make sure that all shapefiles have the exact same fields in the same order, and that they all contain the same geometry type. +### Testing. +- Test PyShp on the Python 3.14 official release (officially released this week, no longer in beta). -### Editing shapefiles +## 3.0.2 -If you need to edit a shapefile you would have to read the -file one record at a time, modify or filter the contents, and write it back out. For instance, to create a copy of a shapefile that only keeps a subset of relevant fields: +### .post1 +- Update version at the top of this Readme file (to make what PyPi users see consistent with the changelog and __version__). - >>> # create writer - >>> w = shapefile.Writer('tests/shapefiles/test/edit') +### .post0 +- Re-release to trigger the deploy job to publish PyShp to Pypi - >>> # define which fields to keep - >>> keep_fields = ['BKG_KEY', 'MEDIANRENT'] +### Bug fix +- Deleted py.typed (probably temporarily). Including the py.typed marker file with a single file package caused type checkers to type check all other libraries installed in the same directory (whether they're typed or not, enforcing type checking for all the adjacent dirs libraries, regardless of the user's intentions for the type checker) (pointed out by @dl1jbe - thanks Thomas). Discussions will be started about a longer term fix, possibly e.g. possibly refactoring to a package, and restoring py.typed. - >>> # copy over the relevant fields from the reader - >>> r = shapefile.Reader("tests/shapefiles/blockgroups") - >>> for field in r.fields[1:]: - ... if field[0] in keep_fields: - ... w.field(*field) +### Code quality: +- Ruff check's UP rule added (mimicks PyUpgrade) (@mwtoews). - >>> # write only the relevant attribute values - >>> for shapeRec in r.iterShapeRecords(fields=keep_fields): - ... w.record(*shapeRec.record) - ... w.shape(shapeRec.shape) - >>> # close writer - >>> w.close() +## 3.0.1 -## 3D and Other Geometry Types +### Improvements +- Reader(shp=, dbf=, shx=) now support pathlib.Paths, and any pathlike object (@mwtoews). +### Bug fixes +- PyShp 3 no longer modifies the global doctest module (@JamesParrott). +### Code quality +- isort replaced by Ruff check's I rule (@mwtoews). +- mypy --strict used in CI (@JamesParrott). +- LICENSE.TXT re-encoded in UTF-8 (@musicinmybrain). -Most shapefiles store conventional 2D points, lines, or polygons. But the shapefile format is also capable -of storing various other types of geometries as well, including complex 3D surfaces and objects. +## 3.0.0 -### Shapefiles with measurement (M) values +### Breaking Changes: +- Python 2 and Python 3.8 support dropped. +- Field info tuple is now a namedtuple (Field) instead of a list. +- Field type codes are now FieldType 'enum' members. +- bbox, mbox and zbox attributes are all new Namedtuples. +- Writer does not mutate Shapes. +- New custom subclasses for each shape type: Null, Multipatch, Point, Polyline, + Multipoint, and Polygon, plus the latter 4's M and Z variants (Reader and + Writer are still compatible with their base class, Shape, as before). +- Shape sub classes are creatable from, and serializable to bytes streams, + as per the shapefile spec. -Measured shape types are shapes that include a measurement value at each vertex, for instance -speed measurements from a GPS device. Shapes with measurement (M) values are added with the following -methods: "pointm", "multipointm", "linem", and "polygonm". The M-values are specified by adding a -third M value to each XY coordinate. Missing or unobserved M-values are specified with a None value, -or by simply omitting the third M-coordinate. +### Improvements: +- Speeded up writing shapefiles by up to another ~27% (on top of the recent ~39% improvement in 2.4.1). +### Code quality +- Statically typed, and checked with Mypy +- Checked with Ruff. +- f-strings +- Remove Python 2 specific functions. +- Run doctests against wheels. +- Testing of wheels before publishing them +- pyproject.toml src layout +- Slow test marked. - >>> w = shapefile.Writer('tests/shapefiles/test/linem') - >>> w.field('name', 'C') +## 2.4.1 - >>> w.linem([ - ... [[1,5,0],[5,5],[5,1,3],[3,3,None],[1,1,0]], # line with one omitted and one missing M-value - ... [[3,2],[2,6]] # line without any M-values - ... ]) +### Improvements: +- Speed up writing shapefiles by up to ~39%. Combined for loops of calls to f.write(pack(...)), into single calls. - >>> w.record('linem1') +### Breaking Change. Support for Python 2 and Pythons <= 3.8 to be dropped. +- PyShp 2.4.1 is the latest (and likely last) version of PyShp to support Python 2.7 and Pythons <= 3.8. +These CPython versions have reached [end of life](https://devguide.python.org/versions/#versions). +- Future development will focus on PyShp v3.0.0 onwards (currently intended to supporting Pythons >= 3.9). +- This will not break any projects, as pip and other package managers should not install PyShp 3.0.0 +(after its release) in unsupported Pythons. But we no longer promise such projects will get PyShp's latest +bug fixes and features. +- If this negatively impacts your project, all feedback about this decision is welcome +on our [the discussion page](https://github.com/GeospatialPython/pyshp/discussions/290). - >>> w.close() +## 2.4.0 -Shapefiles containing M-values can be examined in several ways: +### New Features: +- Reader.iterRecords now allows start and stop to be specified, to lookup smaller ranges of records. +- Equality comparisons between Records now also require the fields to be the same (and in the same order). - >>> r = shapefile.Reader('tests/shapefiles/test/linem') +### Development: +- Code quality tools (Ruff format) run on PyShp +- Network, non-network, or all doctests selectable via command line args +- Network tests made runnable on localhost. - >>> r.mbox # the lower and upper bound of M-values in the shapefile - MBox(mmin=0.0, mmax=3.0) +## 2.3.1 - >>> r.shape(0).m # flat list of M-values - [0.0, None, 3.0, None, 0.0, None, None] +### Bug fixes: - >>> r.close() +- Fix recently introduced issue where Reader/Writer closes file-like objects provided by user (#244) +## 2.3.0 -### Shapefiles with elevation (Z) values +### New Features: -Elevation shape types are shapes that include an elevation value at each vertex, for instance elevation from a GPS device. -Shapes with elevation (Z) values are added with the following methods: "pointz", "multipointz", "linez", and "polyz". -The Z-values are specified by adding a third Z value to each XY coordinate. Z-values do not support the concept of missing data, -but if you omit the third Z-coordinate it will default to 0. Note that Z-type shapes also support measurement (M) values added -as a fourth M-coordinate. This too is optional. +- Added support for pathlib and path-like shapefile filepaths (@mwtoews). +- Allow reading individual file extensions via filepaths. +### Improvements: - >>> w = shapefile.Writer('tests/shapefiles/test/linez') - >>> w.field('name', 'C') +- Simplified setup and deployment (@mwtoews) +- Faster shape access when missing shx file +- Switch to named logger (see #240) - >>> w.linez([ - ... [[1,5,18],[5,5,20],[5,1,22],[3,3],[1,1]], # line with some omitted Z-values - ... [[3,2],[2,6]], # line without any Z-values - ... [[3,2,15,0],[2,6,13,3],[1,9,14,2]] # line with both Z- and M-values - ... ]) +### Bug fixes: - >>> w.record('linez1') +- More robust handling of corrupt shapefiles (fixes #235) +- Fix errors when writing to individual file-handles (fixes #237) +- Revert previous decision to enforce geojson output ring orientation (detailed explanation at https://github.com/SciTools/cartopy/issues/2012) +- Fix test issues in environments without network access (@sebastic, @musicinmybrain). - >>> w.close() +## 2.2.0 -To examine a Z-type shapefile you can do: +### New Features: - >>> r = shapefile.Reader('tests/shapefiles/test/linez') +- Read shapefiles directly from zipfiles. +- Read shapefiles directly from urls. +- Allow fast extraction of only a subset of dbf fields through a `fields` arg. +- Allow fast filtering which shapes to read from the file through a `bbox` arg. - >>> r.zbox # the lower and upper bound of Z-values in the shapefile - ZBox(zmin=0.0, zmax=22.0) +### Improvements: - >>> r.shape(0).z # flat list of Z-values - [18.0, 20.0, 22.0, 0.0, 0.0, 0.0, 0.0, 15.0, 13.0, 14.0] +- More examples and restructuring of README. +- More informative Shape to geojson warnings (see #219). +- Add shapefile.VERBOSE flag to control warnings verbosity (default True). +- Shape object information when calling repr(). +- Faster ring orientation checks, enforce geojson output ring orientation. - >>> r.shape(0).points_3D # list of 3D coordinates incorporating the Z-values - [(1.0, 5.0, 18.0), (5.0, 5.0, 20.0), (5.0, 1.0, 22.0), (3.0, 3.0, 0.0), (1.0, 1.0, 0.0), (3.0, 2.0, 0.0), (2.0, 6.0, 0.0), (3.0, 2.0, 15.0), (2.0, 6.0, 13.0), (1.0, 9.0, 14.0)] +### Bug fixes: - >>> r.close() +- Remove null-padding at end of some record character fields. +- Fix dbf writing error when the number of record list or dict entries didn't match the number of fields. +- Handle rare garbage collection issue after deepcopy (https://github.com/mattijn/topojson/issues/120) +- Fix bug where records and shapes would be assigned incorrect record number (@karanrn) +- Fix typos in docs (@timgates) -### 3D MultiPatch Shapefiles +## 2.1.3 -Multipatch shapes are useful for storing composite 3-Dimensional objects. -A MultiPatch shape represents a 3D object made up of one or more surface parts. -Each surface in "parts" is defined by a list of XYZM values (Z and M values optional), and its corresponding type is -given in the "partTypes" argument. The part type decides how the coordinate sequence is to be interpreted, and can be one -of the following module constants: TRIANGLE_STRIP, TRIANGLE_FAN, OUTER_RING, INNER_RING, FIRST_RING, or RING. -For instance, a TRIANGLE_STRIP may be used to represent the walls of a building, combined with a TRIANGLE_FAN to represent -its roof: +### Bug fixes: - >>> from shapefile import TRIANGLE_STRIP, TRIANGLE_FAN +- Fix recent bug in geojson hole-in-polygon checking (see #205) +- Misc fixes to allow geo interface dump to json (eg dates as strings) +- Handle additional dbf date null values, and return faulty dates as unicode (see #187) +- Add writer target typecheck +- Fix bugs to allow reading shp/shx/dbf separately +- Allow delayed shapefile loading by passing no args +- Fix error with writing empty z/m shapefile (@mcuprjak) +- Fix signed_area() so ignores z/m coords +- Enforce writing the 11th field name character as null-terminator (only first 10 are used) +- Minor README fixes +- Added more tests - >>> w = shapefile.Writer('tests/shapefiles/test/multipatch') - >>> w.field('name', 'C') +## 2.1.2 - >>> w.multipatch([ - ... [[0,0,0],[0,0,3],[5,0,0],[5,0,3],[5,5,0],[5,5,3],[0,5,0],[0,5,3],[0,0,0],[0,0,3]], # TRIANGLE_STRIP for house walls - ... [[2.5,2.5,5],[0,0,3],[5,0,3],[5,5,3],[0,5,3],[0,0,3]], # TRIANGLE_FAN for pointed house roof - ... ], - ... partTypes=[TRIANGLE_STRIP, TRIANGLE_FAN]) # one type for each part +### Bug fixes: - >>> w.record('house1') +- Fix issue where warnings.simplefilter('always') changes global warning behavior [see #203] - >>> w.close() +## 2.1.1 -For an introduction to the various multipatch part types and examples of how to create 3D MultiPatch objects see [this -ESRI White Paper](http://downloads.esri.com/support/whitepapers/ao_/J9749_MultiPatch_Geometry_Type.pdf). +### Improvements: +- Handle shapes with no coords and represent as geojson with no coords (GeoJSON null-equivalent) +- Expand testing to Python 3.6, 3.7, 3.8 and PyPy; drop 3.3 and 3.4 [@mwtoews] +- Added pytest testing [@jmoujaes] +### Bug fixes: -# Testing +- Fix incorrect geo interface handling of multipolygons with complex exterior-hole relations [see #202] +- Enforce shapefile requirement of at least one field, to avoid writing invalid shapefiles [@Jonty] +- Fix Reader geo interface including DeletionFlag field in feature properties [@nnseva] +- Fix polygons not being auto closed, which was accidentally dropped +- Fix error for null geometries in feature geojson +- Misc docstring cleanup [@fiveham] -The testing framework is pytest, and the tests are located in ./tests. -This includes an extensive set of unit tests of the various pyshp features in ./tests/test_shapefile.py, -and tests against various input data. -In the same folder as README.md and pyproject.toml, from the command line run: +## 2.1.0 -```shell -python -m pytest -``` +### New Features: -Additionally, all the code and examples located in this file, README.md, -is tested and verified with the builtin doctest framework. -A special doctest runner will run the doctests, with test artefacts confined to a -a temporary directory, and optionally switching the remote URLs for localhost: +- Added back read/write support for unicode field names. +- Improved Record representation +- More support for geojson on Reader, ShapeRecord, ShapeRecords, and shapes() -```shell -python tests/run_doctests.py -``` +### Bug fixes: +- Fixed error when reading optional m-values +- Fixed Record attribute autocomplete in Python 3 +- Misc readme cleanup -## Network tests +## 2.0.0 -Some of the tests and doctests, are intended to test reading shapefiles from -remote servers, which requires internet connectivity. The pytest tests are marked "network". -For rapid iteration, in CI, or when developing in offline testing environments, these -tests can be dealt with in two ways: - i) by skipping the network tests via : -```shell -pytest -m "not network" -``` -or the doctests via: -```shell -python tests/run_doctests.py -m "not network" -``` -or ii) by cloning a repo of the files they download, serving these on localhost in a separate process, -and running the network tests with the environment variable REPLACE_REMOTE_URLS_WITH_LOCALHOST to `yes`: -Setup a local file server (*): -``` -git clone http://github.com/JamesParrott/PyShp_test_shapefile -cd PyShp_test_shapefile -python -m http.server 8000 -``` -and then: -```bash -REPLACE_REMOTE_URLS_WITH_LOCALHOST=yes && pytest -``` -or the doctests via: -```bash -REPLACE_REMOTE_URLS_WITH_LOCALHOST=yes && python tests/run_doctests.py -``` -The network tests alone can also be run (without also running all the tests that don't -make network requests) using: `pytest -m network` (or the doctests using: `python tests/run_doctests.py -m network`). +The newest version of PyShp, version 2.0 introduced some major new improvements. +A great thanks to all who have contributed code and raised issues, and for everyone's +patience and understanding during the transition period. +Some of the new changes are incompatible with previous versions. +Users of the previous version 1.x should therefore take note of the following changes +(Note: Some contributor attributions may be missing): +### Major Changes: +- Full support for unicode text, with custom encoding, and exception handling. + - Means that the Reader returns unicode, and the Writer accepts unicode. +- PyShp has been simplified to a pure input-output library using the Reader and Writer classes, dropping the Editor class. +- Switched to a new streaming approach when writing files, keeping memory-usage at a minimum: + - Specify filepath/destination and text encoding when creating the Writer. + - The file is written incrementally with each call to shape/record. + - Adding shapes is now done using dedicated methods for each shapetype. +- Reading shapefiles is now more convenient: + - Shapefiles can be opened using the context manager, and files are properly closed. + - Shapefiles can be iterated, have a length, and supports the geo interface. + - New ways of inspecting shapefile metadata by printing. [@megies] + - More convenient accessing of Record values as attributes. [@philippkraft] + - More convenient shape type name checking. [@megies] +- Add more support and documentation for MultiPatch 3D shapes. +- The Reader "elevation" and "measure" attributes now renamed "zbox" and "mbox", to make it clear they refer to the min/max values. +- Better documentation of previously unclear aspects, such as field types. -# Contributors +### Important Fixes: -``` -Atle Frenvik Sveen -Bas Couwenberg -Ben Beasley -Casey Meisenzahl -Charles Arnold -David A. Riggs -davidh-ssec -Edward Kawas -Evan Heidtmann -ezcitron -Felix Oesterle -fiveham -geospatialpython -Hannes -Ignacio Martinez Vazquez -James Parrott -Jason Moujaes -Jonty Wareing -Karim Bahgat -karanrn -Kurt Schwehr -Kyle Kelley -Lionel Guez -Louis Tiao -Marcin Cuprjak -mcuprjak -Micah Cochran -Michael Davis -Michal Čihař -Mike Toews -Miroslav Šedivý -Nilo -pakoun -Paula Spannring -Paulo Ernesto -Raynor Vliegendhart -Razzi Abuissa -RosBer97 -Ross Rogers -Ryan Brideau -Thomas Beierlein -Tim Gates -Tobias Megies -Tommi Penttinen -Uli Köhler -Vsevolod Novikov -Zac Miller -``` +- More reliable/robust: + - Fixed shapefile bbox error for empty or point type shapefiles. [@mcuprjak] + - Reading and writing Z and M type shapes is now more robust, fixing many errors, and has been added to the documentation. [@ShinNoNoir] + - Improved parsing of field value types, fixed errors and made more flexible. + - Fixed bug when writing shapefiles with datefield and date values earlier than 1900 [@megies] +- Fix some geo interface errors, including checking polygon directions. +- Bug fixes for reading from case sensitive file names, individual files separately, and from file-like objects. [@gastoneb, @kb003308, @erickskb] +- Enforce maximum field limit. [@mwtoews] \ No newline at end of file