Skip to content

Commit dfc0107

Browse files
martinv13claude
martinv13
and
claude
authored
Fixes and docs (#73)
* Fix inserted rows count to avoid returning 0 when the count is not returned by the backend * Add zooming capability in `xml2db serve` * Write documentation for CLI usage * Several documentation editing --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent aab9b86 commit dfc0107

11 files changed

Lines changed: 398 additions & 135 deletions

File tree

‎README.md‎

Lines changed: 52 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,64 @@
11
# Loading XML files into a relational database
22

3-
`xml2db` is a Python package which allows parsing and loading XML files into a relational database. It handles complex
3+
`xml2db` is a Python package that parses and loads XML files into a relational database. It handles complex
44
XML files which cannot be denormalized to flat tables, and works out of the box, without any custom mapping rules.
55

6-
It can be used within an [Extract, Load, Transform](https://docs.getdbt.com/terms/elt) data pipeline pattern as it
7-
allows loading XML files into a relational data model which is very close from the source data, yet easy to work with.
6+
It fits naturally into an [Extract, Load, Transform](https://docs.getdbt.com/terms/elt) pipeline: it
7+
loads XML files into a relational data model that stays close to the source data while remaining easy to query as
8+
flat database tables. The raw data can then be transformed using [DBT](https://www.getdbt.com/), SQL views, or stored procedures to produce
9+
more user-friendly tables.
810

911
Starting from an XSD schema which represents a given XML structure, `xml2db` builds a data model, i.e. a set of database
1012
tables linked to each other by foreign keys relationships. Then, it allows parsing and loading XML files into the
1113
database, and getting them back from the database into XML format if needed.
1214

15+
This package uses `sqlalchemy` to interact with the database, so it should work with different database backends.
16+
Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional
17+
packages to connect to your database (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for
18+
MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB).
19+
20+
**Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!**
21+
22+
## Installation
23+
24+
The package can be installed, preferably in a virtual environment, using `pip`:
25+
26+
``` bash
27+
pip install xml2db
28+
```
29+
30+
## CLI
31+
32+
After installation, `xml2db` is available as a command-line tool with three subcommands.
33+
34+
Explore your XSD schema and configure the data model interactively in a browser:
35+
36+
```bash
37+
xml2db serve path/to/schema.xsd
38+
```
39+
40+
This opens a page with an Entity Relationship Diagram, source/target tree views, DDL output, and a live YAML config
41+
editor with autocomplete.
42+
43+
Import an XML file directly from the command line:
44+
45+
```bash
46+
xml2db import file.xml schema.xsd \
47+
--connection-string "postgresql+psycopg2://user:pw@host/db" \
48+
--config model_config.yml
49+
```
50+
51+
Render the ERD, trees, or DDL to stdout or a file without starting a server:
52+
53+
```bash
54+
xml2db render schema.xsd --format erd
55+
xml2db render schema.xsd --format ddl --db-type postgresql
56+
```
57+
58+
See the [CLI reference](https://cre-dev.github.io/xml2db/cli/) for all options.
59+
60+
## Python API
61+
1362
Loading XML files into a relational database with `xml2db` can be as simple as:
1463

1564
```python
@@ -28,28 +77,6 @@ document = data_model.parse_xml(
2877
document.insert_into_target_tables()
2978
```
3079

31-
The data model created by `xml2db` will be close to the XSD schema. However, `xml2db` will perform a few systematic
32-
simplifications aimed at limiting the complexity of the resulting data model and the storage footprint. The resulting
33-
data model can be configured, but the above code will work out of the box, with reasonable defaults.
34-
35-
The raw data loaded into the database can then be processed if need be, using for instance [DBT](https://www.getdbt.com/),
36-
SQL views or stored procedures aimed at extracting, correcting and formatting the data into more user-friendly tables.
37-
38-
This package uses `sqlalchemy` to interact with the database, so it should work with different database backends.
39-
Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional
40-
packages to connect to your database (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for
41-
MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB).
42-
43-
**Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!**
44-
45-
## Installation
46-
47-
The package can be installed, preferably in a virtual environment, using `pip`:
48-
49-
``` bash
50-
pip install xml2db
51-
```
52-
5380
## Testing
5481

5582
Running the tests requires installing additional development dependencies, after cloning the repo, with:

‎docs/api/overview.md‎

Lines changed: 1 addition & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ for lower level steps. It can be useful for advanced use cases, for instance:
3838

3939
* transforming the data in intermediate steps,
4040
* adding logging,
41-
* limiting concurrent access to the database within a multiprocess setup, etc.
41+
* limiting concurrent access to the database in a [multiprocessing context](../how_it_works.md#multiprocessing), etc.
4242

4343
For those scenarios you can easily reimplement
4444
[`Document.insert_into_target_tables`](document.md/#xml2db.document.Document.insert_into_target_tables) to suit your
@@ -59,74 +59,6 @@ flowchart TB
5959
end
6060
```
6161

62-
### Multiprocessing example
63-
64-
XML parsing is CPU-bound and scales well across processes. Loading into the
65-
database, however, must be coordinated to avoid conflicts on shared tables.
66-
The right level of synchronisation depends on the backend:
67-
68-
* **DuckDB (file-based)**: only one active writer is allowed at a time, so
69-
all database I/O must be serialised.
70-
* **PostgreSQL, MS SQL Server, …**: concurrent writes to *different* temp
71-
tables are safe (each process gets a unique temp-table prefix), but the final
72-
merge into the shared target tables should be serialised.
73-
74-
The simplest approach (and the one shown below) is to serialise the entire
75-
database phase with a `multiprocessing.Lock`, keeping only the parsing step
76-
parallel. This works correctly for all backends.
77-
78-
```python
79-
import multiprocessing
80-
from xml2db import DataModel
81-
82-
83-
def load_one_file(xml_path, xsd_path, connection_string, lock):
84-
# Each process creates its own DataModel with a unique temp_prefix.
85-
model = DataModel(
86-
xsd_file=xsd_path,
87-
connection_string=connection_string,
88-
)
89-
# XML parsing is CPU-bound and runs in parallel across all processes.
90-
doc = model.parse_xml(xml_path)
91-
92-
# Serialise all database I/O across processes.
93-
with lock:
94-
doc.insert_into_target_tables()
95-
model.engine.dispose()
96-
97-
98-
if __name__ == "__main__":
99-
xsd_path = "schema.xsd"
100-
connection_string = "duckdb:///data.duckdb"
101-
xml_files = ["file1.xml", "file2.xml", "file3.xml"]
102-
103-
lock = multiprocessing.Lock()
104-
processes = [
105-
multiprocessing.Process(
106-
target=load_one_file,
107-
args=(xml_path, xsd_path, connection_string, lock),
108-
)
109-
for xml_path in xml_files
110-
]
111-
for p in processes:
112-
p.start()
113-
for p in processes:
114-
p.join()
115-
if p.exitcode != 0:
116-
raise RuntimeError(f"Worker failed with exit code {p.exitcode}")
117-
```
118-
119-
!!! Note
120-
For backends that support concurrent writers, you can increase throughput
121-
by splitting
122-
[`Document.insert_into_target_tables`](document.md/#xml2db.document.Document.insert_into_target_tables)
123-
into separate calls to
124-
[`Document.insert_into_temp_tables`](document.md/#xml2db.document.Document.insert_into_temp_tables)
125-
(run concurrently, since each process has a unique temp-table prefix, so
126-
there are no collisions) and
127-
[`Document.merge_into_target_tables`](document.md/#xml2db.document.Document.merge_into_target_tables)
128-
(serialised via lock).
129-
13062
## *Advanced use:* get data from the database back to XML
13163

13264
The flow chart below presents data conversions used to get back data from the database into XML, showing the functions

‎docs/cli.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
---
2+
title: "CLI usage"
3+
description: "Reference for the xml2db command-line interface: import XML files, render ERDs and DDL, and launch the interactive browser explorer."
4+
---
5+
6+
# CLI usage
7+
8+
The `xml2db` CLI provides three subcommands: `import`, `render`, and `serve`.
9+
10+
## xml2db import
11+
12+
Parse an XML file and load it into a database.
13+
14+
```
15+
xml2db import XML_FILE XSD_FILE --connection-string DSN [options]
16+
```
17+
18+
**Positional arguments:**
19+
20+
| Argument | Description |
21+
|---|---|
22+
| `XML_FILE` | Path to the XML file to import |
23+
| `XSD_FILE` | Path to the XSD schema file |
24+
25+
**Options:**
26+
27+
| Option | Description |
28+
|---|---|
29+
| `--connection-string DSN`, `-d DSN` | SQLAlchemy connection string (required) |
30+
| `--config FILE`, `-c FILE` | YAML model config file |
31+
| `--db-schema SCHEMA` | Database schema to use |
32+
| `--metadata KEY=VALUE`, `-m KEY=VALUE` | Metadata values for `metadata_columns` (repeatable) |
33+
| `--short-name NAME` | Data model short name (default: `DocumentRoot`) |
34+
| `--no-iterparse` | Use the recursive parser instead of iterparse (higher memory usage) |
35+
| `--recover` | Attempt to parse malformed XML |
36+
| `--validate` | Validate the XML against the schema before importing |
37+
38+
**Example:**
39+
40+
```bash
41+
xml2db import file.xml schema.xsd \
42+
--connection-string "postgresql+psycopg2://user:pw@host/db" \
43+
--config model_config.yml \
44+
--metadata source=file.xml
45+
```
46+
47+
On success, the command prints the number of rows inserted and already-existing (deduplicated), with per-phase timings.
48+
49+
## xml2db render
50+
51+
Print an ERD, source/target tree, or DDL to stdout or a file, without starting a server.
52+
53+
```
54+
xml2db render XSD_FILE [options]
55+
```
56+
57+
**Positional arguments:**
58+
59+
| Argument | Description |
60+
|---|---|
61+
| `XSD_FILE` | Path to the XSD schema file |
62+
63+
**Options:**
64+
65+
| Option | Description |
66+
|---|---|
67+
| `--config FILE`, `-c FILE` | YAML model config file |
68+
| `--db-names` | Use physical database identifiers in the ERD instead of logical names |
69+
| `--db-type BACKEND` | Database backend for DDL output (`postgresql`, `mssql`, `mysql`, ...) |
70+
| `--format FORMAT`, `-f FORMAT` | Output format: `erd` (default), `target-tree`, `source-tree`, or `ddl` |
71+
| `--output FILE`, `-o FILE` | Write output to a file instead of stdout |
72+
| `--short-name NAME` | Data model short name (default: `DocumentRoot`) |
73+
74+
**Examples:**
75+
76+
```bash
77+
xml2db render schema.xsd --format erd
78+
xml2db render schema.xsd --format target-tree
79+
xml2db render schema.xsd --format source-tree
80+
xml2db render schema.xsd --format ddl --db-type postgresql
81+
xml2db render schema.xsd --format erd --output diagram.md
82+
```
83+
84+
## xml2db serve
85+
86+
Launch an interactive schema explorer in the browser.
87+
88+
```
89+
xml2db serve XSD_FILE [options]
90+
```
91+
92+
The explorer shows four tabs: ERD, target tree, source tree, and DDL. The left panel is a YAML config editor with autocomplete for table names, field names, and all config options. Edits trigger an automatic rebuild. The **Save** button writes the config back to disk.
93+
94+
**Positional arguments:**
95+
96+
| Argument | Description |
97+
|---|---|
98+
| `XSD_FILE` | Path to the XSD schema file |
99+
100+
**Options:**
101+
102+
| Option | Description |
103+
|---|---|
104+
| `--config FILE`, `-c FILE` | YAML model config file to load on startup; Save writes it back to this path (default: `model_config.yml`) |
105+
| `--db-type BACKEND` | Database backend for the DDL tab (`postgresql`, `mssql`, `mysql`, ...) |
106+
| `--no-browser` | Do not open the browser automatically |
107+
| `--port PORT`, `-p PORT` | HTTP port (default: `8765`) |
108+
| `--short-name NAME` | Data model short name (default: `DocumentRoot`) |
109+
110+
**Example:**
111+
112+
```bash
113+
xml2db serve schema.xsd --config model_config.yml --db-type postgresql
114+
```
115+
116+
See [Getting started](getting_started.md) for a walkthrough of the explorer.

‎docs/configuring.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,9 @@ model_config = load_config("model_config.yml")
8080

8181
The following options can be passed as top-level keys of the model configuration `dict`:
8282

83+
* `as_columnstore` (`bool`): for MS SQL Server, create clustered columnstore indexes on all tables. This can be also set up at
84+
the table level for each table. However, for `n-n` relationships tables, this option is the only way to configure the
85+
clustered columnstore indexes. The default value is `False` (disabled).
8386
* `document_tree_hook` (`Callable`): sets a hook function which can modify the data extracted from the XML. It gives direct
8487
access to the underlying tree data structure just before it is extracted to be loaded to the database. This can be used,
8588
for instance, to prune or modify some parts of the document tree before loading it into the database. The document tree
@@ -89,24 +92,21 @@ the declarative [`"transform": "skip"`](#skipping-fields) option is simpler.
8992
similar with `document_tree_hook`, but it is called as soon as a node is completed, not waiting for the entire parsing to
9093
finish. It is especially useful if you intend to filter out some nodes and reduce memory footprint while parsing. For
9194
straightforward field exclusion, see [`"transform": "skip"`](#skipping-fields).
92-
* `row_numbers` (`bool`): adds `xml2db_row_number` columns either to `n-n` relationships tables, or directly to data tables when
93-
deduplication of rows is opted out. This allows recording the original order of elements in the source XML, which is not
94-
always respected otherwise. It was implemented primarily for round-trip tests, but could serve other purposes. The
95-
default value is `False` (disabled).
96-
* `as_columnstore` (`bool`): for MS SQL Server, create clustered columnstore indexes on all tables. This can be also set up at
97-
the table level for each table. However, for `n-n` relationships tables, this option is the only way to configure the
98-
clustered columnstore indexes. The default value is `False` (disabled).
9995
* `metadata_columns` (`list`): a list of extra columns that you want to add to the root table of your model. This is
10096
useful for instance to add the name of the file which has been parsed, or a timestamp, etc. Columns should be specified
10197
as dicts, the only required keys are `name` and `type` (a SQLAlchemy type object); other keys will be passed directly
10298
as keyword arguments to `sqlalchemy.Column`. Actual values need to be passed to
10399
[`DataModel.parse_xml`](api/data_model.md#xml2db.model.DataModel.parse_xml) for each
104100
parsed documents, as a `dict`, using the `metadata` argument.
105-
* `transform` (`false` or `"auto"`): set to `false` to disable all automatic field transformations globally: no joining of multi-value columns, no elevation of child tables, no collapsing of choice groups. The default `"auto"` applies all of these where applicable. Per-field `transform` and per-table `choice_transform` still override the global setting.
106101
* `record_hash_column_name`: the column name to use to store records hash data (defaults to `xml2db_record_hash`).
107102
* `record_hash_constructor`: a function used to build a hash, with a signature similar to `hashlib` constructor
108103
functions (defaults to `hashlib.sha1`).
109104
* `record_hash_size`: the byte size of the record hash (defaults to 20, which is the size of a `sha-1` hash).
105+
* `row_numbers` (`bool`): adds `xml2db_row_number` columns either to `n-n` relationships tables, or directly to data tables when
106+
deduplication of rows is opted out. This allows recording the original order of elements in the source XML, which is not
107+
always respected otherwise. It was implemented primarily for round-trip tests, but could serve other purposes. The
108+
default value is `False` (disabled).
109+
* `transform` (`false` or `"auto"`): set to `false` to disable all automatic field transformations globally: no joining of multi-value columns, no elevation of child tables, no collapsing of choice groups. The default `"auto"` applies all of these where applicable. Per-field `transform` and per-table `choice_transform` still override the global setting.
110110

111111
## Fields configuration
112112

‎docs/getting_started.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,8 @@ This opens a browser with four tabs:
3939
- **Source tree**: a text tree of the raw XSD structure before simplification
4040
- **DDL**: `CREATE TABLE` statements for the target schema
4141

42-
The left panel is a YAML config editor with autocomplete for table names, field names, and all config options. Edit the config and the diagram updates automatically (with a short debounce). When the config looks right, click **Save** to write it to a file (default: `model_config.yml`).
42+
The left panel is a YAML config editor with autocomplete for table names, field names, and all config options. Edit the config and the diagram updates automatically.
43+
When the config looks right, click **Save** to write it to a file (default: `model_config.yml`).
4344

4445
You can also render these representations directly to stdout or a file without the browser:
4546

@@ -54,7 +55,7 @@ See [Configuring your data model](configuring.md) for a full description of the
5455

5556
## Importing XML files
5657

57-
Once the data model looks right, import an XML file into the database:
58+
Once you are happy with the data model, import an XML file into the database (config is optional):
5859

5960
``` bash
6061
xml2db import file.xml schema.xsd \
@@ -96,7 +97,7 @@ with open("data_model_erd.md", "w") as f:
9697
f.write(data_model.get_entity_rel_diagram())
9798
```
9899

99-
The diagram uses [Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html). PyCharm and GitHub both render Mermaid natively.
100+
The diagram uses [Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html). Your IDE should be able to render Mermaid preview.
100101

101102
``` py title="Write source and target trees to files" linenums="1"
102103
with open("source_tree.txt", "w") as f:

0 commit comments

Comments
 (0)