[CodeKitty-max]

Description

Why

ExecuTorch can already run on Espressif chips: examples/espressif/executor_runner
runs .pte programs with the portable CPU kernels. But no backend targets
Espressif's own neural-network kernels, so every operator runs as generic C
on the CPU. ESP-NN is Espressif's int8
kernel library, with SIMD kernels for the ESP32-S3. TFLite Micro uses it, but
ExecuTorch users can't reach it today.

This PR adds EspNNBackend. It is a delegate that lowers PT2E-quantized int8
subgraphs to ESP-NN, while everything else stays on ExecuTorch's portable
kernels. The goal is that the usual ExecuTorch flow (export, quantize, lower,
.pte) produces programs that run on an ESP32-S3 at ESP-NN speed. Models that
don't fully map to ESP-NN still run, because unsupported parts fall back to
the CPU.

What's included

backends/espressif/

• EspNNQuantizer: PT2E annotations for the int8 scheme ESP-NN executes.
  Activations are per-tensor and weights are per-channel symmetric.
• EspNNPartitioner: delegates int8 regions and leaves the rest of the graph
  for the portable kernels. It returns a report of delegated and CPU
  operators, and allow_cpu_fallback=False turns leftovers into an error.
• EspNNBackend.preprocess(): serializes each partition to a compact int8
  graph format (XTS3) that the device runtime consumes directly.
• to_espnn_edge(): a one-call wrapper around to_edge_transform_and_lower.
• espnn_runtime/: an ESP-IDF component. It provides the BackendInterface
  implementation, the XTS3 interpreter on top of ESP-NN, and an optional
  ExecuTorch PAL (esp_timer clock and PSRAM-aware allocation).
• reference/: a bit-exact host model of the device arithmetic, used by the
  tests and by the on-device checks.
• test/: 41 pytest tests.

Delegated operators:

• convolution (strided, depthwise, dilated, transposed) and linear
• relu/relu6/hardtanh, plus table-lookup activations (hardswish, sigmoid,
  silu, tanh, gelu and others)
• add and mul with broadcasting, softmax
• pooling and mean
• nearest and bilinear upsample
• data movement: reshape, permute, slice, pad, cat, split and similar

Partitions can have multiple inputs and outputs.

examples/espressif/, alongside the existing executor_runner/, which is
unchanged:

• quantize_and_lower.py: a minimal export → quantize → lower → .pte
  example.
• aot_espnn_compiler.py: a compiler CLI. It calibrates on real data, can run
  optional QAT, and reports accuracy. It also writes test bundles for the
  board.
• espnn_runner/: an ESP-IDF app that runs bundles on a board and checks
  every output against the host reference before timing it.
• camera_stream/: a live camera classification demo (ESP32-S3).
• scripts/:
  • the ExecuTorch cross-build for ESP-IDF;
  • build, flash and verify on a connected board;
  • benchmarks.
• benchmarks/: measured results for the ESP32-S3 and ESP32.

Usage

from executorch.backends.espressif import EspNNQuantizer, to_espnn_edge
from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e

prepared = prepare_pt2e(torch.export.export(model, example_inputs).module(), EspNNQuantizer())
# ... calibrate ...
edge, report = to_espnn_edge(torch.export.export(convert_pt2e(prepared), example_inputs))
open("model.pte", "wb").write(edge.to_executorch().buffer)

MobileNetV2 lowers to 1 partition with 101 delegated operators and none left
on the CPU.

Test plan

Host, no hardware needed:

pytest backends/espressif/test   # 41 passed
lintrunner --paths-cmd "git ls-files backends/espressif examples/espressif"   # no lint issues

The tests lower small models for every operator group. They check the
serialized graphs bit-exactly against the reference interpreter and against
the quantized PyTorch model, and they run CPU-fallback programs through the
ExecuTorch Python runtime.

On hardware, with an ESP-IDF v5.1+ shell and the board on /dev/ttyACM0:

python examples/espressif/scripts/make_test_bundles.py
python examples/espressif/scripts/run_bundle.py --port /dev/ttyACM0 \
    examples/espressif/espnn_runner/bundles/op_suite.xtb
examples/espressif/scripts/build_executorch_esp.sh esp32s3
python examples/espressif/scripts/run_bundle.py --port /dev/ttyACM0 \
    --executorch-build-dir cmake-out-esp-esp32s3 \
    examples/espressif/espnn_runner/bundles/pte_suite.xtp

Results on real boards:

• ESP32-S3 (N16R8): 11/11 standalone operator cases are bit-exact.
  7/7 ExecuTorch programs pass, including the CPU-fallback cases (LSTM,
  cumsum, a CPU island between two delegates, and a transformer).
• ESP32 (D0WDQ6, no PSRAM): 11/11 operator cases are bit-exact.

Benchmarks (ESP32-S3, 240 MHz, octal PSRAM)

Each model is verified against the host reference on the board before it is
timed. Full tables, including the ESP32, are in examples/espressif/benchmarks/.

Model	Input	Latency
Keyword-spotting DS-CNN	49×10 MFCC	16.4 ms
MobileNetV3-Small	224×224	582 ms
MobileNetV2	224×224	1.61 s
EfficientNet-B0	224×224	2.50 s
LR-ASPP MobileNetV3 segmentation (.pte, fully delegated)	224×224	3.58 s
Small ViT (.pte, 11 ops delegated, 30 on CPU)	64×64	85 ms (22 ms delegate, 63 ms CPU)

Design notes for reviewers

• Memory. The delegate handle comes from the runtime allocator. Each
  delegate allocates its activation arena, scratch buffers and optional PSRAM
  weight copy at init() with ESP-IDF's heap_caps, so it can choose between
  internal SRAM and PSRAM. Nothing is allocated during execute(). I'm happy
  to route this through the runtime allocator instead if that's preferred.
• Device format. The payload is a small custom binary format (XTS3)
  rather than a FlatBuffer. That keeps the device side free of extra
  dependencies, and every region and record is bounds-checked on load.
• ESP-NN dependency. ESP-NN (Apache-2.0) is pulled from the Espressif
  component registry. It is not vendored.

Open questions

1. CI. backends/espressif/test is not yet in pytest.ini testpaths.
   Should I add it in this PR? There's also an existing ESP32-S3 emulator job
   (test-espressif.yml). A follow-up could run espnn_runner's operator
   suite there.
2. Example layout. The new examples sit next to the existing
   executor_runner/. I can merge the shared parts, such as the ExecuTorch
   cross-build and the PAL, with the existing runner if maintainers prefer a
   single Espressif example tree.

Release notes: backends

cc @cccclai @nil-is-all @psiddh @AdrianLundell @digantdesai @rascani

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23956

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 25 Awaiting Approval

As of commit fe18533 with merge base a519331 ():

AWAITING APPROVAL - The following workflows need approval before CI can run:

• Build Aarch64 Linux CUDA Wheels (gh)
• Build Aarch64 Linux Wheels (gh)
• Build documentation (gh)
• Build Linux CUDA Wheels (gh)
• Build Linux Wheels (gh)
• Build macOS Wheels (gh)
• Build Presets (gh)
• Build Windows CUDA Wheels (gh)
• Build Windows Wheels (gh)
• Cadence Build & Test (gh)
• Lint (gh)
• periodic (gh)
• pull (gh)
• Test ARM Backend (gh)
• Test CoreML Backend (gh)
• Test Cortex-M Backend (gh)
• Test ESP32-S3 Emulator (gh)
• Test NXP Backend (gh)
• Test QNN Backend (gh)
• Test QNN Windows MSVC build (gh)
• Test Vulkan Backend (gh)
• Test WebGPU Backend (gh)
• Test XNNPACK Backend (gh)
• trunk (gh)
• Windows MSVC Build (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-cla]

Hi @CodeKitty-max!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[digantdesai]

@claude review this PR.

[digantdesai]

@rascani FYI

[claude]

Claude finished @digantdesai's task in 45s —— View job

---

This PR adds a solid ESP-NN delegate, but it needs changes before merge. A grouped-conv bug produces wrong results and reads out of bounds, the device-side XTS3 parser has integer-overflow holes on 32-bit targets, and none of the 41 host tests run in CI.

PR Review: #23956 (click to expand)

Summary

The PR adds an EspNNBackend delegate: a PT2E quantizer, a partitioner, an XTS3 serializer, an ESP-IDF runtime and interpreter on top of ESP-NN, and examples and benchmarks. Several items block merge:

• a grouped-conv correctness and out-of-bounds bug in the device interpreter;
• integer-overflow gaps in the untrusted-payload parser;
• strict mode (allow_cpu_fallback=False) that doesn't actually enforce full delegation;
• no CI coverage.

I found these by reading the code; I did not build or run anything.

Security

• copy_extent_valid can overflow its int64 accumulator (espnn_runtime/src/xts3_graph.cpp:175-186). Each span = (dims[d]-1) * strides[d] can reach about 2^62, and up to 6 of them are added together with no overflow check. A crafted COPY segment with large dims and strides wraps high negative, so it passes validation. run_copy (int8_interpreter.cpp:~1527) then indexes the arena with overflowing int32 arithmetic, giving arbitrary reads and writes. Fix: reject any segment with |span| >= limit before adding it.
• The COPY parameter_size check wraps on 32-bit size_t (xts3_graph.cpp:201-203). COPY_HEADER_SIZE + size_t(segments) * COPY_SEGMENT_SIZE is computed mod 2^32 on ESP32/S3. A segments value chosen so the product wraps to a small number passes the equality check, and the loop at :213-246 then reads far past the payload. Fix: check segments <= (parameter_size - COPY_HEADER_SIZE) / COPY_SEGMENT_SIZE first.
• The UPSAMPLE size check wraps the same way (xts3_graph.cpp:270-271). 12U * size_t(out_h + out_w) wraps when out_h is about 3.6e8, so a 28-byte parameter blob is accepted. The table loop at :276-284 then reads gigabytes past table. Fix: divide instead of multiplying, or cap out_h/out_w.
• Per-tensor constant offsets are never checked for alignment (xts3_graph.cpp:~946). Only the start of the constant region is checked for 16-byte alignment. An int32 bias with an odd constant_offset is read through const int32_t* (int8_interpreter.cpp:~471), and on Xtensa that raises a LoadStoreAlignment panic. The serializer pads correctly, so only malformed blobs are affected; still, the parser's stated goal is to bounds-check everything. Requiring at least 4-byte alignment for INT32 constants would close this.

Fix the parser overflows →

Thread Safety

• ESP-NN scratch buffers are process-global but set per interpreter (int8_interpreter.cpp:~1653-1659, ~297-303). invoke() calls esp_nn_set_{conv,depthwise_conv,mean}_scratch_buf(scratch_), which writes ESP-NN's static globals. If two delegates or programs run execute() on different FreeRTOS tasks or cores, one interpreter can use another's smaller or already-freed scratch buffer. esp_nn_dual_core_enable() is also global and is never reset. Either document a single-invoker requirement or serialize invoke() with a mutex.

API Design

• allow_cpu_fallback=False does not enforce full delegation (partition/espnn_partitioner.py:374-382). A region whose ops are all supported but that _boundary_is_valid rejects only increments rejected and stays on the CPU. Strict mode raises only when no partition forms at all. The docstring says strict mode is "what the standalone XTS3 firmware path requires", so it should also raise when rejected > 0.
• The report is a side channel on the partitioner (espnn_partitioner.py:331, 397-410; lowering.py). to_espnn_edge returns (edge, partitioner.report), and self.report is overwritten on every partition() call. With a multi-method program, only the last method's report survives. No other backend's lowering helper returns a tuple. Consider returning the EdgeProgramManager and exposing the report through logging or a separate utility. Related: EspNNQuantizer.transform_for_annotation sets the undeclared attributes self.rewritten_split_count and fused_activation_count (espnn_quantizer.py:~255), which goes against the repo's no-dynamic-state rule.
• Quantizer extras ignore per-module configs (quantizer/espnn_quantizer.py:258-266). annotate_fused_activations, annotate_unary_and_softmax and propagate_shared_quantization always use global_config and skip the module-name and type filters that XNNPACK applies (xnnpack_quantizer.py:428-454). After set_module_name("classifier", None), that module's conv+relu, softmax and similar ops are still quantized.
• preprocess ignores compile_specs and has an env-var side effect (espnn_preprocess.py:42-55). ESPNN_DELEGATE_OUTPUT silently overwrites one dump file per partition, so only the last partition is kept. Please drop it, or make it a compile spec.

Infrastructure

• CI never runs the tests. backends/espressif/test isn't in pytest.ini testpaths, and the path filters in test-espressif.yml don't cover backends/espressif/**. To answer open question 1: yes, add it in this PR. The C++ interpreter (about 1.8k LOC) also never runs on the host. A host build of int8_interpreter.cpp and xts3_graph.cpp against ESP-NN's ANSI kernels, compared with the Python reference, would have caught the grouped-conv bug below and any serializer/parser drift. Today xts3.py:parse_graph checks much less than the C++ parser, so the round-trip tests only test Python against itself.
• The duplicated PAL breaks the existing runner (espnn_runtime/src/espnn_pal.cpp, Kconfig:19-22). examples/espressif/executor_runner/esp_pal.cpp already defines strong et_pal_* symbols. With ESPNN_PROVIDE_EXECUTORCH_PAL defaulting to y, adding espnn_runtime to that project gives multiple-definition link errors. Please share one PAL; this ties into open question 2.
• The PAL allocator is PSRAM-only (espnn_pal.cpp:52-54). et_pal_allocate uses MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT with no internal-RAM fallback, and the option doesn't depend on SPIRAM. On boards without PSRAM (like the ESP32 D0WDQ6 you tested), every platform allocation returns null, so PlatformMemoryAllocator and the temp-allocator fallback fail at execute time.
• build_executorch_esp.sh duplicates the existing esp-baremetal preset flow (examples/espressif/README.md, .ci/scripts/test_espressif.sh). It introduces a second output-dir convention (cmake-out-esp-<target>, ESPNN_EXECUTORCH_BUILD_DIR versus the existing ET_BUILD_DIR_PATH) and overwrites toolchain flag files. The -mlongcalls fix belongs in the preset or toolchain.
• Reimplemented infrastructure. Each of these copies something that already exists:
  • _passes/fold_batch_norm.py duplicates backends/transforms/fuse_batch_norm_with_conv.py. It only folds direct children of an nn.Sequential, so ResNet's conv1/bn1 attributes are never folded, and it swaps modules with setattr.
  • _passes/remove_dropout.py is unnecessary after eval() export, and it misses Dropout1d/2d/3d and AlphaDropout.
  • DuplicateQuantizationFanoutPass (espnn_partitioner.py:142-177) overlaps exir/backend/canonical_partitioners/duplicate_dequant_node_pass.py. It also duplicates quantize nodes globally, so on CPU fallback a tensor with N consumers is quantized N times.
• No registration: there's no docs page or backends-overview entry, and no CODEOWNERS entry for /backends/espressif or /examples/espressif.

Testing

• Optional dependencies aren't guarded. test_fold_batch_norm.py:16 imports torchvision at module level, and the other test files import torchao and numpy without pytest.importorskip, so a minimal install gets a collection error instead of a skip.
• The tests don't use the shared backends/test/harness Tester or the backends/test/suite flow that the other backends use.

Code Quality

• Grouped conv (1 < groups < in_c) runs as a dense conv on the device (int8_interpreter.cpp:487, :917-944):

  • state.generic is set only when dilation ≠ 1, so nn.Conv2d(16, 16, 3, groups=4) goes to esp_nn_conv_s8.
  • That function loops over the full input_dims.channels and indexes the filter with stride in_channels.
  • The weight is [16, 4, 3, 3] (576 bytes), but the kernel reads 2304 bytes: it runs past the tensor and mixes channels across groups.
  • The validator (xts3_graph.cpp:134-138) accepts groups > 1, and test_int8_lowering.py:106 covers this case, but only against the Python reference, which handles groups correctly.
  • Fix: route groups != 1 to run_generic_conv, which already handles groups.

  Fix this →

• _attach_outputs_to_producers moves nodes between partitions without a convexity check (espnn_partitioner.py:275-303). CapabilityBasedPartitioner separates a producer from its adjacent quantize or activation mostly when merging them would create a cycle, so this move can produce a cyclic partition. fuse_by_partitions then fails instead of falling back.

• Partition decisions use stand-in constants (espnn_partitioner.py:180-197). With FakeTensor constants, StructuralResolver replaces weight scales with 1.0 and bias with 0, but lower_int8_node makes value-dependent checks: the requant multiplier range, bias overflow and symmetric zero points. A valid conv can be wrongly sent to the CPU. Conversely, a bias that overflows passes partitioning, then preprocess raises and the whole lowering fails with no fallback.

• The serializer doesn't enforce runtime limits (serialization/xts3.py:1869-1912). The runtime rejects more than 16 graph inputs or outputs and more than 1024 tensors or ops (xts3_graph.cpp:834-853), and v5 graphs with opcodes above SOFTMAX. The Python side checks only > 0xFFFF. Such graphs lower fine on the host, then fail init() on the board with just a printf. The partitioner should cap partition size, and serialize_graph should validate the opcodes allowed for each version.

• Dim order and shape aren't checked on delegate I/O (espnn_runtime/src/espnn_backend.cpp:68-70, 92-97, 135-141). Only numel() is compared. A channels-last tensor, or one with permuted dims and the same element count, is accepted and silently produces wrong results. Compare sizes() against the recorded shape and reject non-default dim orders.

• hardtanh relu6 fusion reads min_val/max_val only from positional args (espnn_quantizer.py:129-133). If they're passed as kwargs, it falls back to -1/1 and silently misses the fusion.

• Out-of-scope files:

  • examples/espressif/scripts/license_headers.py is a personal header tool with DEFAULT_HOLDER = "Mwaniki Njagi", and lintrunner already covers headers.
  • scripts/board_dashboard.py is a live HTTP dashboard.
  • camera_stream/imagenet_classes.txt duplicates weights.meta["categories"].
  • The committed benchmarks/*.json files are generated output that will go stale.

  Consider dropping these, or moving them out of the PR.

• Hardcoded paths and ports: /dev/ttyACM0 defaults are spread across five scripts, while the existing docs use /dev/ttyUSB0. benchmark.py:20 references ~/data/imagenette2-160, and the viewer.py:19 docstring points to a non-existent tools/camera_viewer.py.

Performance

• init() never calls processed->Free(), even after the interpreter has made its own blob_copy_ (the misaligned and PSRAM-copy paths, int8_interpreter.cpp:222-235, 275-295). In that case the payload is held twice for the delegate's lifetime, which matters on 512 KB SRAM parts.

Recommendation

Request Changes

To unblock merge, fix:

1. the grouped-conv dispatch;
2. the three parser overflows;
3. strict-mode enforcement;
4. the PSRAM-only allocator and PAL symbol clash;
5. adding backends/espressif/test to CI.

On the reimplemented passes, report side channel, duplicate build script and extra example tooling, please discuss with maintainers before the next round. Together they account for a good share of the 17k LOC.

[digantdesai]

Fantastic. Thanks for the contribution. This is a HUGE PR. It may take us some time to review and provide you meaningful feedback.

[CodeKitty-max]

Fantastic. Thanks for the contribution. This is a HUGE PR. It may take us some time to review and provide you meaningful feedback.

Thank you, I hope to work with y'all till I get this up and running and roll out more tests for other boards.

[CodeKitty-max]

Suggested edits by @claude have been made and pushed

[JakeStevens]

There are a few compile passes added, can you please ensure you use existing where applicable? there should already be batch norm fusion iirc, maybe dropout removal as well

[CodeKitty-max]

There are a few compile passes added, can you please ensure you use existing where applicable? there should already be batch norm fusion iirc, maybe dropout removal as well

Oh, I figure the issue is that I am using custom functions for some functions that have already been implemented, correct?

[CodeKitty-max]

3 custom passes deleted in favor of the ones already in use by Executorch

[rascani]

@CodeKitty-max - Thanks for putting this together! It is high on my priority list to do a detailed review on this.

One early design discussion to work through: ExecuTorch backends come in two different styles: delegate backends and operator library backends. Most of our backends are delegate backends like the one you have created here (they have a partitioner and a runtime BackendInterface implementation). Operator libraries don't have those, but instead have passes that lower the aten ops to custom ops backed by an operator library like Cortex-M's CMSIS-NN or Cadence's nnlib. Given their similarity, did you look at creating an operator library backend like those? I know our documentation is really geared towards the delegate style.

Again, huge thanks for putting this together. I'm really excited to dig into it. I just picked up some ESP32-S3 boards to try it out.

[CodeKitty-max]

I started by looking at delegate backends and even when I did the first primitive prototypes and tests, that's what I had in mind before proceeding. Would the other method be preferred here to this?

[rascani]

I started by looking at delegate backends and even when I did the first primitive prototypes and tests, that's what I had in mind before proceeding. Would the other method be preferred here to this?

There are different trade-offs between the two methods. The nice benefit to operator library style is that you don't need to write your own serialization format or interpreter, and instead can just leverage executorch's. The downside is that there's no partitioner, or any opportunity for storing state in the delegate handle (though there are other tricks to adding state).

[CodeKitty-max]

There are different trade-offs between the two methods. The nice benefit to operator library style is that you don't need to write your own serialization format or interpreter, and instead can just leverage executorch's. The downside is that there's no partitioner, or any opportunity for storing state in the delegate handle (though there are other tricks to adding state).

Then I guess it might not be a dramatic difference in performance. As long as this can already work for the two physical boards that I have already tested, I can focus on improving performance progressively. Have you performed tests on the esp32s3 yet?

[rascani]

There are different trade-offs between the two methods. The nice benefit to operator library style is that you don't need to write your own serialization format or interpreter, and instead can just leverage executorch's. The downside is that there's no partitioner, or any opportunity for storing state in the delegate handle (though there are other tricks to adding state).

Then I guess it might not be a dramatic difference in performance. As long as this can already work for the two physical boards that I have already tested, I can focus on improving performance progressively. Have you performed tests on the esp32s3 yet?

Sorry for the delay.

OpLib vs. Delegate Backend

In terms of pure latency, there's probably not a significant difference for fully delegated models, but there likely would be for models with graph breaks. If you need portable kernels to supplement, then you wind up copying tensor data in & out of the delegate boundaries.

The real wins with operator libraries are in binary size and RAM usage (which arguably matter more than latency). The executorch runtime already includes a serialization format that supports custom kernels and an execution loop to iterate through those (delegates must duplicate this). Additionally, you get the full benefit of executorch's memory planning, which handles tensor lifetimes & copy elision.

I think the long term plan here probably needs to move to an operator library design, but I'm flexible on how we get there.

CI Status

The latest ESP32-S3 job (https://github.com/pytorch/executorch/actions/runs/39214352330/job/121758794836) fails after the existing portable smoke test passes. make_test_bundles.py cannot import test_cpu_fallback from the installed package, so the ESP-NN suite never builds or runs.

Testing

It looks like the ESP-NN testing all goes through the XTS3 layer standalone without using the executorch runtime. This doesn't test the integration fully. Since we have the emulator already set up, we should be able to run full e2e tests with it. Eventually, we can also do the backend test suite as well, but probably better for a follow-up.

Examples

We should consolidate the runner with the existing runner, and we can enhance its functionality if there is anything missing. The camera_stream example also should be using the executorch runtime. I'd probably suggest punting that one to a follow-up PR.

Next Steps

As I mentioned above, I'd like to figure out a path forward here. I'm inclined to start developing an operator library backend, but would like to make sure it also works for your use cases as well. I can either start from scratch there, or we can potentially merge this and then convert it. I can't merge as-is though because of CI failures. WDYT?

[CodeKitty-max]

Sorry for the delay.

OpLib vs. Delegate Backend

In terms of pure latency, there's probably not a significant difference for fully delegated models, but there likely would be for models with graph breaks. If you need portable kernels to supplement, then you wind up copying tensor data in & out of the delegate boundaries.

The real wins with operator libraries are in binary size and RAM usage (which arguably matter more than latency). The executorch runtime already includes a serialization format that supports custom kernels and an execution loop to iterate through those (delegates must duplicate this). Additionally, you get the full benefit of executorch's memory planning, which handles tensor lifetimes & copy elision.

I think the long term plan here probably needs to move to an operator library design, but I'm flexible on how we get there.

CI Status

The latest ESP32-S3 job (https://github.com/pytorch/executorch/actions/runs/39214352330/job/121758794836) fails after the existing portable smoke test passes. make_test_bundles.py cannot import test_cpu_fallback from the installed package, so the ESP-NN suite never builds or runs.

Testing

It looks like the ESP-NN testing all goes through the XTS3 layer standalone without using the executorch runtime. This doesn't test the integration fully. Since we have the emulator already set up, we should be able to run full e2e tests with it. Eventually, we can also do the backend test suite as well, but probably better for a follow-up.

Examples

We should consolidate the runner with the existing runner, and we can enhance its functionality if there is anything missing. The camera_stream example also should be using the executorch runtime. I'd probably suggest punting that one to a follow-up PR.

Next Steps

As I mentioned above, I'd like to figure out a path forward here. I'm inclined to start developing an operator library backend, but would like to make sure it also works for your use cases as well. I can either start from scratch there, or we can potentially merge this and then convert it. I can't merge as-is though because of CI failures. WDYT?

I don't think I should take the camera example to another PR because we might just be complicating something that does not need complication, I have made a few edits on the parts that you have spoken off, if we can make sure that it runs well and passes the tests then we can start the porting post-commit as an ongoing project. I think what is best now is to ensure that we can at least have a functional delegate that uses the Executorch runtime.