Repository navigation
test: check that skills name real MCP operations, CLI commands and files - #8
Merged
Merged
Conversation
Skills can now only name MCP operations, CLI commands, skills, anchors and reference files that exist. A committed snapshot of the public MCP and CLI surface, built from the voiceflow/cli release spec, is the source of truth. verify_cli.py also runs help for every CLI command the skills name and checks their flags against the pinned binary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xavidop
approved these changes
Oct 29, 2026
Contributor
Merge activity
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and resulting behavior
Skills can name an MCP operation, a
vfcommand, another skill or a link anchor that does not exist. Nothing caught that before, so an agent following the skill would call a tool that is not there.python3 scripts/validate.pynow fails, with onefile:line: reasonper problem, when a skill:vfcommand path that the pinned CLI does not have;#anchorthat does not exist, or names a skill folder that does not exist;vf_agent_get.The list of real operations and commands is a committed snapshot,
tests/fixtures/voiceflow-surface.json.scripts/refresh_surface.pyrebuilds it from the public voiceflow/cli release spec: the MCP side from the latest release, the CLI side from the pinned version. A refresh with no upstream change is a no-op.scripts/verify_cli.pyalso runs--helpfor every CLI command the skills name, and checks each flag against the pinned binary.Verification
python3 scripts/validate.py,python3 -m unittest discover -s tests -v(97 tests, 1 skipped) andpython3 scripts/build.pypass on this branch.verify_cli.pypassed against the checksum-verified pinnedvf0.287.0. It used help output and the localhost fixture only, with no account calls.voiceflow-telnyx) and typos that must fail (voiceflow-buidl).scan_secrets.pylocally (Gitleaks not installed here; CI runs it). The suite ran on Python 3.14 locally; CI covers 3.12 on three systems.Compatibility and release notes
Contributor tooling only. No customer-visible change, so no CHANGELOG entry. CONTRIBUTING.md explains how to refresh the snapshot.
🤖 Generated with Claude Code