diff --git a/.jscpd.json b/.jscpd.json index e4a2fc7..1657631 100644 --- a/.jscpd.json +++ b/.jscpd.json @@ -4,5 +4,6 @@ "format": ["python"], "gitignore": true, "ignore": ["**/__pycache__/**", "**/.venv/**", "**/*.egg-info/**"], + "ignorePattern": ["from [\\w.]+ import \\([^)]*\\)"], "path": ["project_validator", "tests"] } diff --git a/README.md b/README.md index ceeda93..46bc5b4 100644 --- a/README.md +++ b/README.md @@ -6,13 +6,13 @@ A tool to validate a project's build, CI and tooling against best practices. It recognises C and C++ (CMake or Meson), Python (`pyproject.toml` or `setup.py`), -Rust (Cargo) and Go (Go modules) projects, and judges each language by its own ecosystem's tools: a C++ +Rust (Cargo), Go (Go modules), JavaScript/TypeScript (`package.json` or `deno.json`) JVM (Java and Kotlin, with Maven or Gradle) PHP (Composer), Ruby (Bundler), .NET (C# and F#, with MSBuild) and Swift (SwiftPM or Xcode) projects, and judges each language by its own ecosystem's tools: a C++ project needs branch coverage from gcovr or llvm-cov, a Python project from coverage.py, a Rust project from cargo-llvm-cov. Checks that do not apply to any of the project's languages are reported as `SKIP`. A project using several languages is judged for each of them and gets the worst result. In a project of no recognised language, only the checks that apply to every project run. A C/C++ project is judged by the rules of the -build system it uses (CMake or Meson); one using neither is judged as a CMake project. +build system it uses (CMake or Meson); one using neither is judged as a CMake project. A JVM project's Maven or Gradle build counts as CI configuration when CI runs it (`mvn verify`, `gradle check`), since that is where its plugins are wired in. More ecosystems can be added with plugins (see [Adding an ecosystem](#adding-an-ecosystem)). The validator itself is pure Python and has no build step; it only inspects @@ -60,33 +60,33 @@ Exit codes: `0` everything passed, `1` a check failed or raised an error, `2` us | Check | Applies to | Passes when | |---|---|---| | `ci-present` | all | at least one CI configuration is found | -| `tests-run` | C/C++, Python, Rust, Go | tests exist and CI runs them (`ctest-runs` is accepted as an old name). C/C++ with CMake: `enable_testing()` or `include(CTest)` is called, tests are registered (`add_test`, `*_discover_tests`, ...) and CI runs `ctest`; with `--build`, the project is really configured and built, and `ctest` must list at least one test. C/C++ with Meson: `test()` is called in a `meson.build` and CI runs `meson test`. Python: `test_*.py` / `*_test.py` files run by pytest, unittest, tox or nox. Rust: `#[test]` functions run by `cargo test` or nextest. Go: `Test` functions in `*_test.go` files run by `go test`. Warns when CI never runs the tests | -| `coverage` | C/C++, Python, Rust, Go | CI reports branch coverage (Go's tools measure statements only, so Go needs statement coverage: `go test -cover` or `-coverprofile`). C/C++: gcovr, lcov, genhtml, gcov or llvm-cov branch options; Python: `coverage run --branch`, `--cov-branch` or `branch = true`; Rust: cargo-llvm-cov, cargo-tarpaulin or grcov with `--branch`. Path or MC/DC coverage (`-fcondition-coverage`, `-fcoverage-mcdc`, `--mcdc`, ...) is reported as a bonus | -| `platform-matrix` | C/C++, Python, Rust, Go | CI covers at least 2 operating systems, 2 toolchains (compilers, Python versions, Rust toolchains or Go versions) and 3 OS/toolchain combinations | -| `ci-build-types` | C/C++, Rust | every OS/toolchain combination in CI builds Debug and Release (Rust: debug and release profiles), each with and without sanitizers | -| `compiler-warnings` | C/C++, Python, Rust, Go | a strict warning set is enabled. C/C++: `-Wall -Wextra -Wpedantic -Wshadow -Wconversion`, plus `/W4` and `/permissive-` when MSVC is in CI; Python: warnings are errors (`-W error`, `PYTHONWARNINGS=error`, pytest `filterwarnings = error`); Rust: `cargo clippy`, `-D warnings` and `clippy::pedantic`; Go: `go vet` and staticcheck (golangci-lint runs both). Warns listing what is missing | -| `warnings-as-errors` | C/C++ | `-Werror`, `/WX` or `COMPILE_WARNING_AS_ERROR` (Meson: `werror=true`) is enabled in at least one configuration | +| `tests-run` | C/C++, Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | tests exist and CI runs them (`ctest-runs` is accepted as an old name). C/C++ with CMake: `enable_testing()` or `include(CTest)` is called, tests are registered (`add_test`, `*_discover_tests`, ...) and CI runs `ctest`; with `--build`, the project is really configured and built, and `ctest` must list at least one test. C/C++ with Meson: `test()` is called in a `meson.build` and CI runs `meson test`. Python: `test_*.py` / `*_test.py` files run by pytest, unittest, tox or nox. Rust: `#[test]` functions run by `cargo test` or nextest. Go: `Test` functions in `*_test.go` files run by `go test`. JavaScript/TypeScript: `*.test.*`, `*.spec.*`, `*_test.*` or `__tests__/` files run by jest, vitest, mocha, ava, `node --test`, `deno test`, `bun test`, playwright or `npm test`. JVM: `@Test` methods or Kotest specs under `src/test/java` or `src/test/kotlin`, run by `mvn test` / `verify` or `gradle test` / `check` / `build`. PHP: `*Test.php` files with PHPUnit test cases or Pest tests, run by phpunit, pest, paratest or `composer test`, in CI or the Composer scripts it runs. Ruby: `spec/**/*_spec.rb` or `test/**/*_test.rb` files run by rspec, `rake test` / `spec` or `rails test`, or by the Rakefile's test tasks when CI runs rake. .NET: a test project referencing `Microsoft.NET.Test.Sdk` (or `MSTest.Sdk`) with `[Fact]`, `[Test]` or `[TestMethod]` tests, run by `dotnet test`. Swift: `XCTestCase` classes or Swift Testing `@Test` functions, run by `swift test` or `xcodebuild test`. Warns when CI never runs the tests | +| `coverage` | C/C++, Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | CI reports branch coverage (Go's tools measure statements only, so Go needs statement coverage: `go test -cover` or `-coverprofile`). C/C++: gcovr, lcov, genhtml, gcov or llvm-cov branch options; Python: `coverage run --branch`, `--cov-branch` or `branch = true`; Rust: cargo-llvm-cov, cargo-tarpaulin or grcov with `--branch`; JavaScript/TypeScript: c8, nyc, `jest --coverage` (or `collectCoverage: true`), `vitest --coverage` (or coverage `enabled: true`), `deno test --coverage` or `node --experimental-test-coverage`, in CI, the package scripts it runs, or the test runner's configuration; JVM: JaCoCo (`jacoco-maven-plugin` with its `report` goal, `jacocoTestReport`) or Kover; PHP: PHPUnit `--path-coverage` (or `pathCoverage="true"`), which reports branch and path coverage together, so it also earns the path bonus (line coverage alone counts no branches); Ruby: SimpleCov with `enable_coverage :branch` (in `.simplecov` or a test helper); .NET: coverlet (`XPlat Code Coverage`, `CollectCoverage=true`, also in a `.runsettings` file), `dotnet-coverage collect` or `dotnet test --coverage`. Swift's coverage counts regions, not branches, so Swift needs region coverage: `swift test --enable-code-coverage`, `xcodebuild -enableCodeCoverage YES`, or a scheme's `codeCoverageEnabled`. Path or MC/DC coverage (`-fcondition-coverage`, `-fcoverage-mcdc`, `--mcdc`, ...) is reported as a bonus | +| `platform-matrix` | C/C++, Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | CI covers at least 2 operating systems, 2 toolchains (compilers, Python versions, Rust toolchains, Go versions, Node.js, Deno or Bun versions, JDK versions, PHP versions, Ruby versions and implementations, .NET SDK versions, or Swift or Xcode versions) and 3 OS/toolchain combinations | +| `ci-build-types` | C/C++, Rust, Swift | every OS/toolchain combination in CI builds Debug and Release (Rust: debug and release profiles; Swift: SwiftPM's `-c release` / debug default, or xcodebuild's `-configuration`), each with and without sanitizers | +| `compiler-warnings` | C/C++, Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | a strict warning set is enabled. C/C++: `-Wall -Wextra -Wpedantic -Wshadow -Wconversion`, plus `/W4` and `/permissive-` when MSVC is in CI; Python: warnings are errors (`-W error`, `PYTHONWARNINGS=error`, pytest `filterwarnings = error`); Rust: `cargo clippy`, `-D warnings` and `clippy::pedantic`; Go: `go vet` and staticcheck (golangci-lint runs both); JavaScript/TypeScript: a linter runs (eslint, `biome lint` / `biome ci`, or `deno lint`); JVM: javac's `-Xlint:all` (a Kotlin-only project passes on Kotlin's default warnings); PHP: every error is reported while testing (`error_reporting` `-1` or `E_ALL` in `phpunit.xml`, `php.ini` or CI); Ruby: Ruby's warnings are on while testing (`ruby -w`, `RUBYOPT=-w`, `$VERBOSE = true`, RSpec's `--warnings` / `config.warnings`, or a test task's `warning = true`) and RuboCop or Standard runs; .NET: C# projects set `AnalysisMode` (or `AnalysisLevel`) to Recommended or All, `EnforceCodeStyleInBuild` and `Nullable` `enable` in a project file or `Directory.Build.props`, which count when CI builds; F#-only projects turn on more compiler warnings (``, `--warnon:`); Swift: SwiftLint runs and concurrency is checked completely (Swift 6 language mode, `-strict-concurrency=complete`, `StrictConcurrency`, or Xcode's `SWIFT_STRICT_CONCURRENCY = complete`). Warns listing what is missing | +| `warnings-as-errors` | C/C++, JavaScript/TypeScript, JVM, PHP, .NET, Swift | C/C++: `-Werror`, `/WX` or `COMPILE_WARNING_AS_ERROR` (Meson: `werror=true`) is enabled in at least one configuration; JavaScript/TypeScript: a lint warning fails CI (`eslint --max-warnings 0`, `biome ci` or `--error-on-warnings`, or `deno lint`); JVM: `-Werror` for javac or kotlinc, or Kotlin's `allWarningsAsErrors`; PHP: PHPUnit's `failOnWarning`, `failOnNotice` and `failOnDeprecation` (or `failOnAllIssues`); .NET: `TreatWarningsAsErrors` or `-warnaserror`; Swift: `-warnings-as-errors`, SwiftPM's `treatAllWarnings(as: .error)` or Xcode's `SWIFT_TREAT_WARNINGS_AS_ERRORS = YES` | | `cxx-extensions` | C/C++ | `CMAKE_CXX_EXTENSIONS` (or `CMAKE_C_EXTENSIONS` for C) is turned off, or with Meson `cpp_std` (`c_std`) names a standard such as `c++20` rather than `gnu++20`, so `-std=c++NN` is used rather than `-std=gnu++NN` | -| `hardening` | C/C++, Rust, Go | binary hardening is enabled: exploit mitigations that the build inserts into the binary (static analysis and dependency audits are judged by `security-scanning` and `dependency-audit`). C/C++: `_FORTIFY_SOURCE`, `-fstack-protector-strong`, `-fcf-protection`, `-fstack-clash-protection` and `_GLIBCXX_ASSERTIONS` or `_LIBCPP_HARDENING_MODE` (or GCC's `-fhardened`), plus `/sdl /guard:cf /DYNAMICBASE /CETCOMPAT` when MSVC is in CI; Rust: `overflow-checks` in release builds; Go: programs are built as position-independent executables (`-buildmode=pie`, not the default on Linux), and a module with no `package main` is skipped. Python builds no binary, so it is not judged. Warns listing what is missing | -| `code-formatting` | C/C++, Python, Rust, Go | the formatting style is defined (`.clang-format`; ruff or black settings; `rustfmt.toml`) and CI runs the formatter. Python, Rust and Go (gofmt, gofumpt, goimports) formatters have a default style, so CI enforcement alone passes. Warns when the style is defined but CI does not enforce it | -| `function-limits` | C/C++, Python, Rust, Go | function length is limited: lizard runs with a length limit (`lizard --length 30`), or for C/C++, `.clang-tidy` enables `readability-function-size` with a threshold and clang-tidy runs, or for Go, golangci-lint runs with `funlen` enabled | -| `static-analysis` | C/C++, Python, Rust, Go | static analyzers run. C/C++: both clang-tidy (`CMAKE_CXX_CLANG_TIDY`, `run-clang-tidy`, or `clang-tidy` in CI) with its checks chosen in a `.clang-tidy`, and cppcheck (`CMAKE_CXX_CPPCHECK` or `cppcheck` in CI); Python: `ruff check`, pylint or flake8, in CI or as a pre-commit hook; Rust: `cargo clippy`; Go: golangci-lint with linters enabled beyond its defaults (`enable:` or `default: all` in a `.golangci.yml`). Warns when an analyzer is configured but does not run, or runs short of that; fails when none runs | -| `type-checking` | Python | CI (or a pre-commit hook) runs a static type checker: mypy, pyright or basedpyright, pyrefly, or ty. Strict mode is a bonus (`strict = true` in `[tool.mypy]` or a `[mypy]` section, `mypy --strict`, or pyright's `typeCheckingMode` `strict`); without it the result suggests it. Warns when a type checker is configured but does not run; fails when none runs. Compiled languages, whose compilers check types, are not judged | +| `hardening` | C/C++, Rust, Go, Swift | binary hardening is enabled: exploit mitigations that the build inserts into the binary (static analysis and dependency audits are judged by `security-scanning` and `dependency-audit`). C/C++: `_FORTIFY_SOURCE`, `-fstack-protector-strong`, `-fcf-protection`, `-fstack-clash-protection` and `_GLIBCXX_ASSERTIONS` or `_LIBCPP_HARDENING_MODE` (or GCC's `-fhardened`), plus `/sdl /guard:cf /DYNAMICBASE /CETCOMPAT` when MSVC is in CI; Rust: `overflow-checks` in release builds; Go: programs are built as position-independent executables (`-buildmode=pie`, not the default on Linux), and a module with no `package main` is skipped; Swift: releases keep the language's runtime safety checks, so no build uses `-Ounchecked` or unchecked exclusivity. Python, JavaScript/TypeScript, the JVM, PHP and Ruby build no native binary, so they are not judged; .NET is not judged yet. Warns listing what is missing | +| `code-formatting` | C/C++, Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | the formatting style is defined (`.clang-format`; ruff or black settings; `rustfmt.toml`) and CI runs the formatter. Python, Rust, Go (gofmt, gofumpt, goimports) and JavaScript/TypeScript (`prettier --check`, `biome format` / `biome ci`, `deno fmt --check` or `dprint check`, in CI or the package scripts it runs) and JVM (Spotless, fmt-maven-plugin, google-java-format, ktfmt or ktlint) and PHP (`php-cs-fixer check` / `--dry-run`, phpcs or `pint --test`, in CI or the Composer scripts it runs) and Ruby (RuboCop, not `-a`; Standard; or `stree check`) and .NET (`dotnet format --verify-no-changes`, `csharpier check` or `fantomas --check`) and Swift (`swift format lint --strict` or `swiftformat --lint`) formatters have a default style, so CI enforcement alone passes. Warns when the style is defined but CI does not enforce it | +| `function-limits` | C/C++, Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | function length is limited: lizard runs with a length limit (`lizard --length 30`), or for C/C++, `.clang-tidy` enables `readability-function-size` with a threshold and clang-tidy runs, or for Go, golangci-lint runs with `funlen` enabled, or for JavaScript/TypeScript, eslint runs with `max-lines-per-function` enabled, or for the JVM, Checkstyle runs with `MethodLength` or detekt runs with `LongMethod` (on by default), or for PHP, PHPMD runs with its `codesize` rules (`ExcessiveMethodLength`), or for Ruby, RuboCop runs with `Metrics/MethodLength` not disabled (it is on by default). lizard does not measure F#, so F#-only projects are skipped; for Swift, SwiftLint runs with `function_body_length` not disabled (it is on by default) | +| `static-analysis` | C/C++, Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | static analyzers run. C/C++: both clang-tidy (`CMAKE_CXX_CLANG_TIDY`, `run-clang-tidy`, or `clang-tidy` in CI) with its checks chosen in a `.clang-tidy`, and cppcheck (`CMAKE_CXX_CPPCHECK` or `cppcheck` in CI); Python: `ruff check`, pylint or flake8, in CI or as a pre-commit hook; Rust: `cargo clippy`; Go: golangci-lint with linters enabled beyond its defaults (`enable:` or `default: all` in a `.golangci.yml`); JavaScript/TypeScript: eslint, `biome lint` or `deno lint`, in CI, a pre-commit hook or the package scripts CI runs; JVM: Error Prone, SpotBugs, PMD, Checkstyle or detekt; PHP: PHPStan, Psalm, PHPMD or Phan; Ruby: RuboCop, Standard or Reek; .NET: an analyzer package beyond the SDK's own (Roslynator, SonarAnalyzer, StyleCop.Analyzers or Meziantou.Analyzer) or FSharpLint; Swift: SwiftLint. Warns when an analyzer is configured but does not run, or runs short of that; fails when none runs | +| `type-checking` | Python, JavaScript/TypeScript, PHP, Ruby | CI (or a pre-commit hook) runs a static type checker. Python: mypy, pyright or basedpyright, pyrefly, or ty; JavaScript/TypeScript: `tsc` (JavaScript with `checkJs`), or `deno check` / `deno test`, also through the package scripts CI runs; PHP: PHPStan or Psalm, also through the Composer scripts CI runs; Ruby: Sorbet (`srb tc`) or Steep (`steep check`). Strict mode is a bonus (`strict = true` in `[tool.mypy]` or a `[mypy]` section, `mypy --strict`, pyright's `typeCheckingMode` `strict`, `"strict": true` or `@tsconfig/strictest` in `tsconfig.json`, Deno, strict by default, PHPStan `level` 8 or higher, Psalm `errorLevel` 1 or 2, Sorbet's `# typed: strict`, or `D::Ruby.strict` in a Steepfile); without it the result suggests it. Warns when a type checker is configured but does not run; fails when none runs. Compiled languages, whose compilers check types, are not judged | | `action-pinning` | all | every action and reusable workflow CI uses from another repository is pinned to a full commit SHA (`actions/checkout@ # v4.2.2`), and every `docker://` image to a digest. Fails when one is pinned to a branch or not at all; warns when one is pinned to a tag or an abbreviated commit (`--allow-official-tags` accepts tags for `actions/` and `github/`). Skipped without GitHub Actions | | `ci-lint` | all | CI (or a pre-commit hook) lints the configuration of each CI provider it uses: GitHub Actions with actionlint or zizmor, GitLab CI with `glab ci lint` or `gitlab-ci-local --preview`, CircleCI with `circleci config validate`. Warns otherwise; skipped without CI, or when no linter is known for the providers used | | `copy-paste-detection` | all | CI runs a duplicate code detector (PMD CPD, jscpd, Duplo, Simian, CCFinderX or SonarQube); warns otherwise | -| `security-scanning` | all | CI runs code security scanning (CodeQL or GitLab SAST), or every language in the project that has security lints enables them: Python: bandit (or ruff's `S` rules); Rust: `unsafe_code` forbidden crate-wide; Go: gosec. Warns otherwise, naming the languages left unscanned | +| `security-scanning` | all | CI runs code security scanning (CodeQL or GitLab SAST), or every language in the project that has security lints enables them: Python: bandit (or ruff's `S` rules); Rust: `unsafe_code` forbidden crate-wide; Go: gosec; JavaScript/TypeScript: `eslint-plugin-security` (or `@microsoft/eslint-plugin-sdl`), or semgrep with a JavaScript ruleset; JVM: SpotBugs with Find Security Bugs, or PMD's `category/java/security.xml` rules; PHP: Psalm's taint analysis or phpcs-security-audit; Ruby: brakeman, or RuboCop, whose Security cops are on by default; .NET: the analyzers' Security category (`AnalysisModeSecurity`, or `category-Security` severity warning or error in `.editorconfig`), or Security Code Scan. Warns otherwise, naming the languages left unscanned | | `secret-scanning` | all | CI or a pre-commit hook runs a secret scanner: gitleaks, TruffleHog, detect-secrets, ggshield or Nosey Parker. Warns otherwise, and says so when a scanner is configured (`.gitleaks.toml`, `.secrets.baseline`, `.gitguardian.yaml`) but nothing runs it. GitHub's own secret scanning is a repository setting, which cannot be seen from a checkout | -| `dependency-management` | C/C++, Python, Rust, Go | dependencies come from a package manager and are locked. C/C++: CPM, FetchContent, vcpkg (`vcpkg.json`), Conan (`conanfile.txt` / `conanfile.py`) or Meson wraps (`subprojects/*.wrap`); Python: dependencies declared in `pyproject.toml`, `setup.py` or requirements files are locked by a committed `uv.lock`, `poetry.lock`, `pdm.lock`, `Pipfile.lock` or `pylock.toml`, or by requirements pinned with `--hash`; Rust: a binary crate commits `Cargo.lock` (a library need not); Go: a module that requires others commits `go.sum`. Passes with a note when no dependencies are declared. Warns when they are not locked, or when third-party code is committed (`third_party/`, `external/`, `extern/`, `vendor/` other than `go mod vendor` output, or git submodules) | -| `dependency-audit` | Python, Rust, Go | CI checks the dependencies against known vulnerabilities. Python: pip-audit or safety; Rust: cargo audit or `cargo deny check` (advisories); Go: govulncheck. osv-scanner, `trivy fs` and `snyk test` count for every language. Warns otherwise | +| `dependency-management` | C/C++, Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | dependencies come from a package manager and are locked. C/C++: CPM, FetchContent, vcpkg (`vcpkg.json`), Conan (`conanfile.txt` / `conanfile.py`) or Meson wraps (`subprojects/*.wrap`); Python: dependencies declared in `pyproject.toml`, `setup.py` or requirements files are locked by a committed `uv.lock`, `poetry.lock`, `pdm.lock`, `Pipfile.lock` or `pylock.toml`, or by requirements pinned with `--hash`; Rust: a binary crate commits `Cargo.lock` (a library need not); Go: a module that requires others commits `go.sum`; JavaScript/TypeScript: dependencies declared in `package.json` (or `imports` in `deno.json`) are locked by a committed `package-lock.json`, `npm-shrinkwrap.json`, `pnpm-lock.yaml`, `yarn.lock`, `bun.lock`, `bun.lockb` or `deno.lock`, libraries included; JVM: dependencies declared in `pom.xml` or a Gradle build are verified by a committed `gradle/verification-metadata.xml` or `gradle.lockfile`, or Maven runs with `--strict-checksums`; PHP: dependencies required in `composer.json` (besides `php` and extensions) are locked by a committed `composer.lock`, unless the package is a `"type": "library"`; Ruby: gems in the `Gemfile` (or a gemspec's dependencies) are locked by a committed `Gemfile.lock`, unless the project is a gem (has a gemspec); .NET: NuGet `PackageReference`s are locked by committed `packages.lock.json` files; Swift: an application (an Xcode project, or a package with an executable) commits `Package.resolved`. Passes with a note when no dependencies are declared. Warns when they are not locked, or when third-party code is committed (`third_party/`, `external/`, `extern/`, `vendor/` other than `go mod vendor` output, or git submodules) | +| `dependency-audit` | Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | CI checks the dependencies against known vulnerabilities. Python: pip-audit or safety; Rust: cargo audit or `cargo deny check` (advisories); Go: govulncheck; JavaScript/TypeScript: `npm audit` (not `npm audit signatures`), `pnpm audit`, `yarn npm audit` / `yarn audit` or `bun audit`; JVM: OWASP dependency-check (`dependency-check-maven` bound to `check`, or `dependencyCheckAnalyze`); PHP: `composer audit`, or `roave/security-advisories` required; Ruby: bundler-audit; .NET: `NuGetAuditMode` `all` (restoring then audits transitive packages too) or `dotnet list package --vulnerable`; Swift has no audit tool of its own, so only the multi-ecosystem scanners count. osv-scanner, `trivy fs` and `snyk test` count for every language. Warns otherwise | | `data-wellformed` | all | every JSON, TOML, YAML and XML file parses, including the configuration in `.github/`, `.gitlab/`, `.circleci/`, `.buildkite/`, `.devcontainer/` and `.vscode/`. JSON that tools read with comments (`*.jsonc`, `tsconfig*.json`, `jsconfig.json`, `devcontainer.json`, `.vscode/*.json`) may have comments and trailing commas; YAML tags of any tool (GitLab's `!reference`) are accepted; XML external entities are never read. Files over 5 MiB are not parsed. Fails listing each file that does not parse, with the line and column | | `large-files` | all | no committed file is larger than 5 MiB (`--max-file-size`) unless Git LFS stores it (`filter=lfs` in a `.gitattributes`) or DVC tracks it (a `.dvc` file beside it). In a git checkout only the files git tracks are judged. Warns listing the largest files | | `security-policy` | all | a `SECURITY.md` (or `.rst`, `.txt`, ...) at the root, in `.github/` or in `docs/` says how to report a vulnerability: an email address, a link, or the forge's private vulnerability reporting. Warns otherwise | -| `license` | all | a license file (`LICENSE`, `LICENSE.md`, `LICENSE-MIT`, `COPYING`, `UNLICENSE`, ... or REUSE's `LICENSES/` directory) holds a recognised license: an `SPDX-License-Identifier` line, or the standard text of AGPL, LGPL, GPL, Apache-2.0, MPL-2.0, EPL-2.0, BSL-1.0, Unlicense, CC0-1.0, MIT, BSD, ISC, 0BSD or Zlib. Warns when the license is not recognised, or when a manifest declares another one (Python: `[project] license`; Rust: `[package] license`). Fails without a license file | +| `license` | all | a license file (`LICENSE`, `LICENSE.md`, `LICENSE-MIT`, `COPYING`, `UNLICENSE`, ... or REUSE's `LICENSES/` directory) holds a recognised license: an `SPDX-License-Identifier` line, or the standard text of AGPL, LGPL, GPL, Apache-2.0, MPL-2.0, EPL-2.0, BSL-1.0, Unlicense, CC0-1.0, MIT, BSD, ISC, 0BSD or Zlib. Warns when the license is not recognised, or when a manifest declares another one (Python: `[project] license`; Rust: `[package] license`; JavaScript/TypeScript: `license` in `package.json`; PHP: `license` in `composer.json`; Ruby: the gemspec's `license` or `licenses`; .NET: `PackageLicenseExpression`). Fails without a license file | | `dependency-updates` | all | Dependabot or Renovate is configured; warns otherwise | | `status-badges` | all | the README (at the root, in `.github/` or in `docs/`) shows a CI status badge of any supported CI provider (GitHub Actions `badge.svg`, GitLab pipeline, CircleCI, ... or their shields.io badges) and a quality badge (Codecov, Coveralls, SonarCloud, Codacy, Code Climate, DeepSource, Coverity Scan, OpenSSF Scorecard or Best Practices). Warns when only one kind is shown; fails when there is no README or no badge | -| `committed-artifacts` | C/C++, Python, Rust, Go | no build output is committed. C/C++: object files and libraries (`*.o`, `*.obj`, `*.a`, `*.lib`, `*.so`, `*.dylib`, `*.dll`), `*.exe`, `*.pdb`, `CMakeCache.txt`, `CMakeFiles/`, `CMakeUserPresets.json`, and a top-level `build/`, `out/` or `builddir/`; Python: `__pycache__/`, `*.py[co]`, `*.egg-info/`, a top-level `dist/` or `build/`; Rust: a top-level `target/`; Go: `*.test`, `*.out` and a top-level `bin/`. In a git checkout only the files git tracks are judged; `--exclude` keeps fixtures out. Fails listing the outputs | -| `vcs-ignore` | C/C++, Python, Rust, Go | a `.gitignore`, `.hgignore`, `.bzrignore` or `.fossil-settings/ignore-glob` ignores the build outputs (`build/` or `out/`, `__pycache__/` or `*.py[cod]`, `/target`, Go's `*.test` or `*.out`). Warns when an ignore file misses them | +| `committed-artifacts` | C/C++, Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | no build output is committed. C/C++: object files and libraries (`*.o`, `*.obj`, `*.a`, `*.lib`, `*.so`, `*.dylib`, `*.dll`), `*.exe`, `*.pdb`, `CMakeCache.txt`, `CMakeFiles/`, `CMakeUserPresets.json`, and a top-level `build/`, `out/` or `builddir/`; Python: `__pycache__/`, `*.py[co]`, `*.egg-info/`, a top-level `dist/` or `build/`; Rust: a top-level `target/`; Go: `*.test`, `*.out` and a top-level `bin/`; JavaScript/TypeScript: `node_modules/`, a top-level `coverage/` and `*.tsbuildinfo` (not `dist/`, which a JavaScript GitHub Action commits); JVM: `*.class`, plus Maven's `target/` or Gradle's `build/` and `.gradle/`; PHP: a top-level `vendor/`, `.phpunit.cache/` and `.phpunit.result.cache`; Ruby: `.bundle/`, `*.gem`, and a top-level `coverage/` or `pkg/` (a committed `vendor/bundle/` is reported by `dependency-management`); .NET: `obj/`, `*.dll`, `*.pdb` and `*.nupkg`; Swift: SwiftPM's `.build/`, Xcode's `xcuserdata/` and `DerivedData/`. In a git checkout only the files git tracks are judged; `--exclude` keeps fixtures out. Fails listing the outputs | +| `vcs-ignore` | C/C++, Python, Rust, Go, JavaScript/TypeScript, JVM, PHP, Ruby, .NET, Swift | a `.gitignore`, `.hgignore`, `.bzrignore` or `.fossil-settings/ignore-glob` ignores the build outputs (`build/` or `out/`, `__pycache__/` or `*.py[cod]`, `/target`, Go's `*.test` or `*.out`, `node_modules/`, Maven's `target/` or Gradle's `build/`, PHP's `vendor/`, Ruby's `.bundle/` or `vendor/bundle/`, .NET's `bin/` or `obj/`, SwiftPM's `.build/`, Xcode's `xcuserdata/` or `DerivedData/`). Warns when an ignore file misses them | CI detection is pluggable too. GitHub Actions workflows (including `strategy.matrix` with `include`/`exclude`) and GitLab CI pipelines (including `parallel: matrix:`) diff --git a/project_validator/build_configurations.py b/project_validator/build_configurations.py index c4b749e..ba4434b 100644 --- a/project_validator/build_configurations.py +++ b/project_validator/build_configurations.py @@ -1,4 +1,5 @@ -"""Recognising the CMake build types, Cargo profiles and sanitizer settings a CI job builds with.""" +"""Recognising the CMake build types, Cargo profiles, SwiftPM and Xcode configurations and sanitizer settings a CI +job builds with.""" import re from collections.abc import Iterable @@ -20,6 +21,11 @@ _CARGO_RELEASE = re.compile(r"--release\b|--profile[=\s]+release\b") _CARGO_OTHER_PROFILE = re.compile(r"--profile\b") +# SwiftPM builds debug unless told otherwise; xcodebuild is judged only when it names its configuration. +_SWIFT_COMMANDS = re.compile(r"\bswift\s+(?:build|test|run)\b([^\\\n\"&;|]*)") +_SWIFT_CONFIGURATION = re.compile(r"(?:\s-c|--configuration)[=\s]+(debug|release)\b", re.IGNORECASE) +_XCODEBUILD_CONFIGURATION = re.compile(r"\bxcodebuild\b[^\\\n\"&;|]*\s-configuration\s+[\"']?(\w+)", re.IGNORECASE) + _DISABLED = r"(?!(?:off|false|no|none|0)\b)" _SANITIZERS_ENABLED = re.compile( r"[-/]fsanitize=\w" @@ -50,10 +56,19 @@ def _cargo_build_types(texts: Iterable[str]) -> set[str]: return {profile for profile in profiles if profile is not None} +def _swift_build_types(texts: Iterable[str]) -> set[str]: + configurations = ( + _SWIFT_CONFIGURATION.search(match[1]) for text in texts for match in _SWIFT_COMMANDS.finditer(text) + ) + swiftpm = {found[1].lower() if found else "debug" for found in configurations} + xcode = (match[1].lower() for text in texts for match in _XCODEBUILD_CONFIGURATION.finditer(text)) + return swiftpm | {BUILD_TYPE_KINDS[value] for value in xcode if value in BUILD_TYPE_KINDS} + + def detect_build_types(*texts: str) -> set[str]: values = (value for text in texts for match in _BUILD_TYPE_SOURCES.finditer(text) for value in match.groups()) cmake = {BUILD_TYPE_KINDS[value.lower()] for value in values if value and value.lower() in BUILD_TYPE_KINDS} - return cmake | _cargo_build_types(texts) + return cmake | _cargo_build_types(texts) | _swift_build_types(texts) def sanitizers_enabled(texts: Iterable[str]) -> bool: diff --git a/project_validator/checks/code_formatting.py b/project_validator/checks/code_formatting.py index 2911f09..be37ff6 100644 --- a/project_validator/checks/code_formatting.py +++ b/project_validator/checks/code_formatting.py @@ -11,7 +11,7 @@ ) def code_formatting(formatter: Formatter, context: Context) -> Result: config = formatter.config_file(context) - tools = matching_names(context.ci_texts, formatter.runs) + tools = matching_names(formatter.texts(context), formatter.runs) if tools and config is not None: return Result.passed(f"{config} is defined and CI runs {', '.join(tools)}") if tools and formatter.default_style: diff --git a/project_validator/checks/data_wellformed.py b/project_validator/checks/data_wellformed.py index c6497a1..5debd4f 100644 --- a/project_validator/checks/data_wellformed.py +++ b/project_validator/checks/data_wellformed.py @@ -1,7 +1,6 @@ """Every JSON, TOML, YAML and XML file parses: real validation, since a broken data file is a bug.""" import json -import re import tomllib from collections import Counter from collections.abc import Callable, Mapping @@ -11,6 +10,7 @@ from project_validator import yaml_support from project_validator.core import Context, Result, check +from project_validator.text import strip_json_comments # Hidden directories that hold configuration worth parsing (CI, editors, dev containers). CONFIGURATION_DIRECTORIES = frozenset({".github", ".gitlab", ".circleci", ".buildkite", ".devcontainer", ".vscode"}) @@ -33,24 +33,10 @@ def _decoded(data: bytes) -> str: raise _ParseError(f"not UTF-8 ({error.reason} at byte {error.start})") from error -# A string, a comment, or a comma that only whitespace separates from the closing bracket. -_JSONC_TOKENS = re.compile(r'"(?:\\.|[^"\\])*"|//[^\n]*|/\*.*?\*/|,(?=\s*[}\]])', re.DOTALL) - - -def _without_comments(text: str) -> str: - """JSON with comments and trailing commas as plain JSON; comments become blanks, so positions are kept.""" - - def blank(match: re.Match[str]) -> str: - token = match[0] - return token if token.startswith('"') else re.sub(r"[^\n]", " ", token) - - return _JSONC_TOKENS.sub(blank, _JSONC_TOKENS.sub(blank, text)) - - def _parse_json(data: bytes, *, comments: bool = False) -> None: text = _decoded(data) try: - json.loads(_without_comments(text) if comments else text) + json.loads(strip_json_comments(text) if comments else text) except json.JSONDecodeError as error: raise _ParseError(error.msg, error.lineno, error.colno) from error diff --git a/project_validator/checks/dependency_audit.py b/project_validator/checks/dependency_audit.py index c87f7f4..436ba8a 100644 --- a/project_validator/checks/dependency_audit.py +++ b/project_validator/checks/dependency_audit.py @@ -10,6 +10,6 @@ rule=DependencyAudit, ) def dependency_audit(audit: DependencyAudit, context: Context) -> Result: - if tools := matching_names(context.ci_texts, audit.tools): + if tools := matching_names(audit.texts(context), audit.tools): return Result.passed(f"CI audits the dependencies with {', '.join(tools)}") return Result.warned("no dependency vulnerability audit found in CI configuration", audit.hint) diff --git a/project_validator/checks/tests_run.py b/project_validator/checks/tests_run.py index c8e2950..fba88f8 100644 --- a/project_validator/checks/tests_run.py +++ b/project_validator/checks/tests_run.py @@ -12,6 +12,6 @@ ) def tests_run(suite: TestSuite, context: Context) -> Result: result = suite.found(context) - if result.status != Status.PASS or matching_names(context.ci_texts, suite.runs): + if result.status != Status.PASS or matching_names(suite.texts(context), suite.runs): return result return Result.warned(f"{result.message}, but no CI configuration runs {suite.runner}", *result.details) diff --git a/project_validator/checks/vcs_ignore.py b/project_validator/checks/vcs_ignore.py index 7715cca..8545fea 100644 --- a/project_validator/checks/vcs_ignore.py +++ b/project_validator/checks/vcs_ignore.py @@ -4,7 +4,7 @@ from dataclasses import dataclass from project_validator.core import Context, Result -from project_validator.ecosystems import check_each_ecosystem, ecosystem_check, rules_of +from project_validator.ecosystems import ecosystem_check, judge_each_ecosystem from project_validator.rules import BuildOutputs @@ -180,4 +180,4 @@ def vcs_ignore(context: Context) -> Result: f"no version control ignore file found ({IGNORE_FILE_NAMES}); build outputs can be committed by accident", "consider adding a .gitignore (or your version control system's equivalent) that ignores build outputs", ) - return check_each_ecosystem(context, rules_of(BuildOutputs), _outputs_ignored) + return judge_each_ecosystem(context, BuildOutputs, _outputs_ignored) diff --git a/project_validator/ecosystems/__init__.py b/project_validator/ecosystems/__init__.py index 6bc1dcc..5fdf2d0 100644 --- a/project_validator/ecosystems/__init__.py +++ b/project_validator/ecosystems/__init__.py @@ -8,7 +8,10 @@ from project_validator.core import Check, CheckFunction, Context, Result, Status from project_validator.ecosystems.c_cpp import C_AND_CXX +from project_validator.ecosystems.dotnet import DOTNET from project_validator.ecosystems.go import GO +from project_validator.ecosystems.javascript import JAVASCRIPT_AND_TYPESCRIPT +from project_validator.ecosystems.jvm import JVM from project_validator.ecosystems.model import ( C_FAMILY, BuildSystem, @@ -21,10 +24,13 @@ languages_of, toolchain_versions, ) +from project_validator.ecosystems.php import PHP from project_validator.ecosystems.python import PYTHON +from project_validator.ecosystems.ruby import RUBY from project_validator.ecosystems.rust import RUST +from project_validator.ecosystems.swift import SWIFT -BUILT_IN = (C_AND_CXX, PYTHON, RUST, GO) +BUILT_IN = (C_AND_CXX, PYTHON, RUST, GO, JAVASCRIPT_AND_TYPESCRIPT, JVM, PHP, RUBY, DOTNET, SWIFT) _registered: list[Ecosystem] = list(BUILT_IN) @@ -168,6 +174,16 @@ def judge_ecosystem[R]( return Result(status=Status.SKIP, message=f"not applicable to {ecosystem.label} built with {used}") +def judge_each_ecosystem[R](context: Context, kind: type[R], judge: Judge[R]) -> Result: + """`judge` of the rule of type `kind` of each ecosystem the project uses, for the build systems it uses (see + `judge_ecosystem`): the worst result wins, and findings are labelled.""" + structure = context.structure + used = ecosystems_for(structure.languages, providers(kind)) + return combined( + {ecosystem: judge_ecosystem(ecosystem, kind, judge, context, structure.build_systems) for ecosystem in used} + ) + + def practice[R]( *, name: str, description: str, rule: type[R], aliases: tuple[str, ...] = () ) -> Callable[[Judge[R]], Check]: @@ -179,14 +195,7 @@ def practice[R]( def wrap(judge: Judge[R]) -> Check: def run(context: Context) -> Result: - structure = context.structure - used = ecosystems_for(structure.languages, providers(rule)) - return combined( - { - ecosystem: judge_ecosystem(ecosystem, rule, judge, context, structure.build_systems) - for ecosystem in used - } - ) + return judge_each_ecosystem(context, rule, judge) languages = languages_of(providers(rule)) return Check(name=name, description=description, run=run, languages=languages, rule=rule, aliases=aliases) @@ -203,9 +212,15 @@ def is_build_tree(directory: Path) -> bool: "BUILT_IN", "C_AND_CXX", "C_FAMILY", + "DOTNET", "GO", + "JAVASCRIPT_AND_TYPESCRIPT", + "JVM", + "PHP", "PYTHON", + "RUBY", "RUST", + "SWIFT", "BuildSystem", "Ecosystem", "RegistryError", @@ -220,6 +235,7 @@ def is_build_tree(directory: Path) -> bool: "ecosystems_for", "is_build_tree", "isolated_registry", + "judge_each_ecosystem", "judge_ecosystem", "languages_of", "practice", diff --git a/project_validator/ecosystems/dotnet.py b/project_validator/ecosystems/dotnet.py new file mode 100644 index 0000000..5a7014c --- /dev/null +++ b/project_validator/ecosystems/dotnet.py @@ -0,0 +1,302 @@ +""".NET (C# and F#): built with MSBuild and the dotnet CLI, judged by the .NET SDK versions CI runs. + +A .NET build reads its settings (analyzers, warnings, coverage, auditing) from the project files and +Directory.Build.props, so those count as CI configuration when CI builds the project. +""" + +import re +from collections.abc import Callable +from fnmatch import fnmatchcase +from pathlib import Path, PurePosixPath + +from project_validator.core import Context, Result, Status +from project_validator.ecosystems.model import BuildSystem, Ecosystem, ToolchainPatterns, versions_axis +from project_validator.evidence import ci_and_hook_texts, configuration, existing_texts +from project_validator.rules import ( + MULTI_ECOSYSTEM_AUDITS, + SUGGESTED_LIZARD, + Analyzer, + BuildOutputs, + CoverageTools, + DeclaredLicense, + DependencyAudit, + DependencyManagement, + Formatter, + FunctionLimits, + SecurityLints, + StaticAnalysis, + TestSuite, + WarningPolicy, + WarningsAsErrors, + directories, + ignored_at_top_level, +) +from project_validator.text import Pattern, pattern + +CSHARP = "C#" +FSHARP = "F#" +PROJECT_FILES = ("*.csproj", "*.fsproj") +# MSBuild reads these beside (or above) every project. +SHARED_SETTINGS = ("Directory.Build.props", "Directory.Build.targets", "Directory.Packages.props") +SETTINGS_FILES = (*PROJECT_FILES, *SHARED_SETTINGS) + + +def _is_build_output(directory: Path) -> bool: + """`bin/` and `obj/` beside a project file hold its build output.""" + if directory.name not in {"bin", "obj"}: + return False + return any(fnmatchcase(entry.name, glob) for entry in directory.parent.iterdir() for glob in PROJECT_FILES) + + +MSBUILD = BuildSystem( + name="MSBuild", + files=(*PROJECT_FILES, "*.sln", "*.slnx", *SHARED_SETTINGS, "global.json"), + is_build_tree=_is_build_output, +) + +RUNS_BUILD = re.compile(r"\bdotnet\s+(?:build|test|publish|pack|run|restore)\b") + + +def _settings_files(context: Context) -> list[str]: + """The project files and shared MSBuild settings, in every directory.""" + files = context.project.files() + return [path for path in files if any(fnmatchcase(PurePosixPath(path).name, glob) for glob in SETTINGS_FILES)] + + +def _build_texts(context: Context) -> tuple[str, ...]: + """CI configuration and pre-commit hooks, with the project files and shared MSBuild settings when CI builds.""" + texts = ci_and_hook_texts(context) + if not any(map(RUNS_BUILD.search, texts)): + return texts + return (*texts, *(context.project.read_text(path) for path in _settings_files(context))) + + +def _coverage_texts(context: Context) -> tuple[str, ...]: + """The build's texts, and the .runsettings files that configure test runs.""" + settings = [path for path in context.project.files() if path.endswith(".runsettings")] + return (*_build_texts(context), *(context.project.read_text(path) for path in settings)) + + +def _with_files(*files: str) -> Callable[[Context], tuple[str, ...]]: + def texts(context: Context) -> tuple[str, ...]: + return (*_build_texts(context), *existing_texts(context, files)) + + return texts + + +def _property(name: str, values: str) -> str: + """An MSBuild property set to one of `values`, in a project file or on the command line.""" + return rf"<{name}>\s*(?:{values})\s*|[-/]p(?:roperty)?:{name}=(?:{values})\b" + + +# Each of them has a default style: dotnet format reads .editorconfig, and CSharpier and Fantomas have their own. +FORMATTERS = Formatter( + name="dotnet format, CSharpier or Fantomas", + config_file=configuration((".editorconfig", ".csharpierrc", ".csharpierrc.json", ".csharpierrc.yaml")), + runs=( + pattern("dotnet format", r"\bdotnet\s+format\b[^\n]*--verify-no-changes\b"), + pattern("CSharpier", r"\bcsharpier\s+(?:check\b|[^\n]*--check\b)"), + pattern("Fantomas", r"\bfantomas\b[^\n]*--check\b"), + ), + hint="consider a CI job that runs dotnet format --verify-no-changes (or csharpier check, or fantomas --check)", + default_style=True, +) + +TEST_SDK = re.compile(r"\bMicrosoft\.NET\.Test\.Sdk\b|\bSdk\s*=\s*[\"']MSTest\.Sdk\b") +# xUnit, NUnit and MSTest, in C# ([Fact]) and F# ([]). +TEST_ATTRIBUTE = re.compile(r"\[ Result: + projects = [path for path in _settings_files(context) if path.endswith(("proj", ".props"))] + if not any(TEST_SDK.search(context.project.read_text(path)) for path in projects): + return Result.failed("no .NET test project found (one referencing Microsoft.NET.Test.Sdk)") + sources = (path for path in context.project.files() if path.endswith(SOURCE_SUFFIXES)) + count = sum(1 for path in sources if TEST_ATTRIBUTE.search(context.project.read_text(path))) + if count == 0: + return Result.failed("no .NET tests found ([Fact], [Test] or [TestMethod] methods)") + return Result.passed(f".NET tests found in {count} file(s)") + + +TESTS = TestSuite(found=_dotnet_tests, runs=(pattern("dotnet test", r"\bdotnet\s+test\b"),), runner="dotnet test") + +# coverlet, Microsoft's dotnet-coverage and Microsoft.Testing.Platform's coverage all count branches. +COVERAGE = CoverageTools( + searched="CI configuration, the MSBuild settings it builds with, or a .runsettings file", + texts=_coverage_texts, + branch=( + pattern("coverlet", rf"XPlat Code Coverage|{_property('CollectCoverage', 'true')}"), + pattern("dotnet-coverage", r"\bdotnet-coverage\s+collect\b"), + pattern("Microsoft.Testing.Platform --coverage", r"\bdotnet\s+test\b[^\n]*\s--coverage\b"), + ), +) + +ANALYSIS_MODE = Pattern( + "AnalysisMode Recommended or All", + re.compile( + r"\s*(?:Recommended|All)\s*" + r"|\s*[\w.]*-?(?:recommended|all)\s*", + re.IGNORECASE, + ), +) +ENFORCE_CODE_STYLE = Pattern("EnforceCodeStyleInBuild", re.compile(_property("EnforceCodeStyleInBuild", "true"), re.I)) +NULLABLE = Pattern("Nullable enable", re.compile(_property("Nullable", "enable"), re.IGNORECASE)) +# The .NET analyzers do not analyze F#; its compiler has warnings to turn on. +FSHARP_WARNINGS = Pattern("F# --warnon", re.compile(r"[^<]+|--warnon:\S+", re.IGNORECASE)) + + +def _required_settings(context: Context) -> tuple[Pattern, ...]: + if CSHARP in context.structure.languages: + return (ANALYSIS_MODE, ENFORCE_CODE_STYLE, NULLABLE) + return (FSHARP_WARNINGS,) + + +WARNINGS = WarningPolicy( + kind=".NET analyzer settings", + recognised=(ANALYSIS_MODE, ENFORCE_CODE_STYLE, NULLABLE, FSHARP_WARNINGS), + required=_required_settings, + texts=_build_texts, + separator=", ", +) + +WARNINGS_AS_ERRORS = WarningsAsErrors( + flags=( + pattern( + "TreatWarningsAsErrors", + rf"{_property('TreatWarningsAsErrors', 'true')}|(?true in Directory.Build.props", +) + +DEPENDENCY_AUDIT = DependencyAudit( + tools=( + pattern("NuGet audit of every package", r"\s*all\s*"), + pattern( + "dotnet list package --vulnerable", + r"\bdotnet\s+(?:list\s+(?:\S+\s+)?package|package\s+list)\b[^\n]*--vulnerable\b", + ), + *MULTI_ECOSYSTEM_AUDITS, + ), + hint="set all, so restoring audits transitive packages too, or run " + "dotnet list package --vulnerable --include-transitive in CI", + texts=_build_texts, +) + +ANALYZER_CONFIGURATION = (".editorconfig", ".globalconfig") + +SECURITY_LINTS = SecurityLints( + texts=_with_files(*ANALYZER_CONFIGURATION), + lints=( + pattern( + ".NET security analyzers", + r"\s*(?:Recommended|All)\s*" + r"|\bdotnet_analyzer_diagnostic\.category-Security\.severity\s*=\s*(?:warning|error)\b", + ), + pattern("Security Code Scan", r"\bSecurityCodeScan\b"), + ), +) + + +def _fsharp_only(context: Context) -> Result: + """lizard measures C# but not F#, and no F# tool is known to limit function length.""" + if CSHARP in context.structure.languages: + return Result.failed(f"function length is never limited; run {SUGGESTED_LIZARD} in CI") + return Result(status=Status.SKIP, message="no known tool limits the length of F# functions") + + +def _package(name: str) -> re.Pattern[str]: + return re.compile(rf"""Include\s*=\s*["']{re.escape(name)}\b""", re.IGNORECASE) + + +STATIC_ANALYSIS = StaticAnalysis( + analyzers=( + Analyzer(name="Roslynator", runs=_package("Roslynator")), + Analyzer(name="SonarAnalyzer", runs=_package("SonarAnalyzer")), + Analyzer(name="StyleCop.Analyzers", runs=_package("StyleCop.Analyzers")), + Analyzer(name="Meziantou.Analyzer", runs=_package("Meziantou.Analyzer")), + Analyzer(name="FSharpLint", runs=re.compile(r"\bfsharplint\s+lint\b", re.IGNORECASE)), + ), + texts=_build_texts, + hint="consider an analyzer package beyond the SDK's own (Roslynator, SonarAnalyzer, StyleCop.Analyzers or " + "Meziantou.Analyzer; FSharpLint for F#)", +) + +PACKAGE_REFERENCE = re.compile(r" str | None: + texts = (context.project.read_text(path) for path in _settings_files(context)) + return "managed with NuGet" if any(map(PACKAGE_REFERENCE.search, texts)) else None + + +def _lockfiles(context: Context) -> str | None: + locks = [path for path in context.tracked_files if PurePosixPath(path).name == LOCKFILE] + if not locks: + return None + return locks[0] if len(locks) == 1 else f"{len(locks)} {LOCKFILE} files" + + +DEPENDENCIES = DependencyManagement( + managed=_uses_packages, + locked=_lockfiles, + lock_hint="set true, commit the packages.lock.json " + "files, and restore them in CI with --locked-mode", +) + +PACKAGE_LICENSE = re.compile(r"\s*([^<]+?)\s*") + + +def _package_license(context: Context) -> str | None: + texts = (context.project.read_text(path) for path in _settings_files(context)) + return next((match[1] for text in texts if (match := PACKAGE_LICENSE.search(text))), None) + + +LICENSE = DeclaredLicense(manifest="the project's PackageLicenseExpression", declared=_package_license) + +BUILD_OUTPUT = BuildOutputs( + name="bin/ and obj/", + globs=ignored_at_top_level(r"\[Bb\]in|\[Oo\]bj|bin|obj"), + paths=directories("bin", "obj"), + hint="consider ignoring bin/ and obj/ (the dotnet new gitignore template does)", + # bin/ is a common name for scripts too; obj/ and compiled assemblies are .NET's own. + artifacts=("obj/", "*.dll", "*.pdb", "*.nupkg"), +) + +_SDK_VERSION = r"(\d+\.\d+(?:\.[\dx]+)?|\d+\.x)" + +DOTNET = Ecosystem( + label=".NET", + languages=frozenset({CSHARP, FSHARP}), + build_systems=(MSBUILD,), + source_suffixes={".cs": CSHARP, ".fs": FSHARP, ".fsi": FSHARP}, + # F# scripts and Razor views belong to .NET projects, but name no language on their own. + ambiguous_suffixes={".fsx": None, ".razor": None, ".cshtml": None}, + toolchain=ToolchainPatterns( + language=".NET SDK", + versions=( + re.compile(rf"\bdotnet-version\W+{_SDK_VERSION}"), + re.compile(rf"\bmcr\.microsoft\.com/dotnet/sdk:{_SDK_VERSION}"), + ), + usage=re.compile(r"\bdotnet\s+(?:build|test|restore|publish|pack|run|format)\b|\bsetup-dotnet\b"), + ), + rules=( + versions_axis(".NET SDK version", ".NET SDK"), + FORMATTERS, + COVERAGE, + TESTS, + WARNINGS, + WARNINGS_AS_ERRORS, + SECURITY_LINTS, + DEPENDENCY_AUDIT, + FunctionLimits(without_lizard=_fsharp_only), + STATIC_ANALYSIS, + DEPENDENCIES, + BUILD_OUTPUT, + LICENSE, + ), +) diff --git a/project_validator/ecosystems/javascript.py b/project_validator/ecosystems/javascript.py new file mode 100644 index 0000000..1b91d6a --- /dev/null +++ b/project_validator/ecosystems/javascript.py @@ -0,0 +1,412 @@ +"""JavaScript and TypeScript: packaged with package.json (npm, pnpm, yarn, bun) or deno.json, judged by the Node.js, +Deno and Bun versions CI runs. + +Much of a JavaScript project's tooling runs through the scripts of its package.json (`npm run lint`), so the scripts +CI runs count as CI configuration wherever a rule reads more than CI configuration. +""" + +import json +import re +from collections.abc import Iterator + +from project_validator.core import Context, Result +from project_validator.ecosystems.model import BuildSystem, Ecosystem, ToolchainPatterns, versions_axis +from project_validator.evidence import ( + ci_and_hook_texts, + configuration, + existing_texts, + first_existing, + json_value, + scripts_run, +) +from project_validator.rules import ( + MULTI_ECOSYSTEM_AUDITS, + SUGGESTED_LIZARD, + Analyzer, + BuildOutputs, + CoverageTools, + DeclaredLicense, + DependencyAudit, + DependencyManagement, + Formatter, + FunctionLimits, + SecurityLints, + StaticAnalysis, + TestSuite, + TypeChecking, + WarningPolicy, + WarningsAsErrors, + directories, + ignored_at_top_level, +) +from project_validator.text import Pattern, matching_names, pattern + +JAVASCRIPT = "JavaScript" +TYPESCRIPT = "TypeScript" +PACKAGE_JSON = "package.json" +DENO_CONFIG_FILES = ("deno.json", "deno.jsonc") +# A package.json that names what it provides is a JavaScript package, even before any source is found. +ENTRY_POINTS = ("main", "module", "exports", "bin") + + +def _declared_by_package(text: str) -> frozenset[str]: + """package.json is everywhere, as the tooling of projects in other languages; only a package's entry points (or + JavaScript sources, recognised by their suffixes) make a project a JavaScript one.""" + try: + manifest = json.loads(text) + except json.JSONDecodeError: + return frozenset() + is_package = isinstance(manifest, dict) and any(key in manifest for key in ENTRY_POINTS) + return frozenset({JAVASCRIPT}) if is_package else frozenset() + + +NPM = BuildSystem(name="npm", files=(PACKAGE_JSON,), declared_languages=_declared_by_package) +DENO = BuildSystem(name="Deno", files=DENO_CONFIG_FILES, languages=frozenset({TYPESCRIPT})) + + +def _json_strings(value: object) -> dict[str, str]: + return {key: item for key, item in value.items() if isinstance(item, str)} if isinstance(value, dict) else {} + + +def _scripts(context: Context) -> dict[str, str]: + """The package.json scripts and deno.json tasks, by name.""" + tasks = (json_value(context, path, "tasks", comments=True) for path in DENO_CONFIG_FILES) + found = {name: script for value in tasks for name, script in _json_strings(value).items()} + return {**found, **_json_strings(json_value(context, PACKAGE_JSON, "scripts"))} + + +# `npm test`, `npm run lint`, `pnpm lint`, `yarn run lint`, `bun run lint`, `deno task lint`. +RUNS_SCRIPT = re.compile( + r"\bnpm\s+(?:run(?:-script)?\s+(?P[\w:.-]+)|(?Ptest|t|start)\b)" + r"|\b(?:pnpm|yarn)\s+(?:run\s+)?(?P[\w:.-]+)" + r"|\bbun\s+run\s+(?P[\w:.-]+)" + r"|\bdeno\s+task\s+(?P[\w:.-]+)" +) +NPM_LIFECYCLE_ALIASES = {"t": "test"} + + +def _named_scripts(text: str) -> Iterator[str]: + for match in RUNS_SCRIPT.finditer(text): + name = next(group for group in match.groups() if group) + yield NPM_LIFECYCLE_ALIASES.get(name, name) + + +def _pre_and_post(name: str) -> tuple[str, ...]: + return (f"pre{name}", f"post{name}") + + +def _run_texts(context: Context) -> tuple[str, ...]: + """CI configuration, pre-commit hooks, and the package scripts they run.""" + texts = ci_and_hook_texts(context) + return (*texts, *scripts_run(texts, _scripts(context), _named_scripts, _pre_and_post)) + + +PRETTIER_CONFIG_FILES = tuple( + f"{stem}{suffix}" + for stem, suffixes in ( + (".prettierrc", ("", ".json", ".json5", ".yaml", ".yml", ".toml", ".js", ".cjs", ".mjs", ".ts")), + ("prettier.config", (".js", ".cjs", ".mjs", ".ts", ".cts", ".mts")), + ) + for suffix in suffixes +) +FORMATTER_CONFIG_FILES = (*PRETTIER_CONFIG_FILES, "biome.json", "biome.jsonc", "dprint.json", ".dprint.json") + + +def _formatter_config(context: Context) -> str | None: + if found := first_existing(context, FORMATTER_CONFIG_FILES): + return found + return PACKAGE_JSON if json_value(context, PACKAGE_JSON, "prettier") is not None else None + + +# Every one of them has a default style, so a configuration file is not needed. +FORMATTERS = Formatter( + name="prettier, biome, deno fmt or dprint", + config_file=_formatter_config, + runs=( + pattern("prettier", r"\bprettier\b[^\n]*\s(?:--check|-c|--list-different|-l)\b"), + pattern("biome", r"\bbiome\s+(?:format|check|ci)\b"), + pattern("deno fmt", r"\bdeno\s+fmt\b[^\n]*\s--check\b"), + pattern("dprint", r"\bdprint\s+check\b"), + ), + hint="consider a CI job that runs prettier --check . (or biome ci, deno fmt --check or dprint check)", + default_style=True, + texts=_run_texts, +) + +JS_SUFFIXES = (".js", ".mjs", ".cjs", ".jsx") +TS_SUFFIXES = (".ts", ".mts", ".cts", ".tsx") +SOURCE_SUFFIXES = (*JS_SUFFIXES, *TS_SUFFIXES) +# app.test.ts, app.spec.js, app_test.ts (Deno), or any source in a __tests__ directory (Jest). +TEST_FILE = re.compile(r"(?:^|/)(?:[^/]*(?:[._]test|\.spec)\.[^/]+|__tests__/.+)$") +RUNS_TESTS = ( + pattern( + "JavaScript tests", + r"\b(?:jest|vitest|mocha|ava|playwright\s+test)\b|\bnode\s+(?:[^\n]*\s)?--test\b|\b(?:deno|bun)\s+test\b" + r"|\bnpm\s+(?:t|test|run\s+test)\b|\b(?:pnpm|yarn)\s+(?:run\s+)?test\b", + ), +) + + +def _javascript_tests(context: Context) -> Result: + tests = [path for path in context.project.files() if path.endswith(SOURCE_SUFFIXES) and TEST_FILE.search(path)] + if not tests: + return Result.failed("no JavaScript or TypeScript tests found (*.test.*, *.spec.* or __tests__/ files)") + return Result.passed(f"JavaScript or TypeScript tests found in {len(tests)} file(s)") + + +TESTS = TestSuite( + found=_javascript_tests, + runs=RUNS_TESTS, + runner="jest, vitest, mocha, node --test, deno test, bun test or playwright (or npm test)", + texts=_run_texts, +) + +# Istanbul (nyc, jest, vitest's istanbul provider) and V8 coverage (c8, vitest, node, deno) both report branches. +COVERAGE_CONFIG_FILES = tuple( + f"{tool}.config.{suffix}" for tool in ("jest", "vitest", "vite") for suffix in ("js", "cjs", "mjs", "ts", "mts") +) + + +def _coverage_texts(context: Context) -> tuple[str, ...]: + return (*_run_texts(context), *existing_texts(context, (PACKAGE_JSON, *COVERAGE_CONFIG_FILES))) + + +COVERAGE = CoverageTools( + searched="CI configuration, the package scripts it runs, or the test runner's settings", + texts=_coverage_texts, + branch=( + pattern("c8", r"(? tuple[Pattern, ...]: + """One linter is enough: the ones that run, or eslint when none does.""" + texts = _run_texts(context) + return tuple(linter for linter in LINTERS if matching_names(texts, (linter,))) or LINTERS[:1] + + +# JavaScript has no compiler warnings; its linters report what other compilers warn about. +WARNINGS = WarningPolicy( + kind="JavaScript linters", recognised=LINTERS, required=_linters_required, texts=_run_texts, separator=", " +) + +WARNINGS_AS_ERRORS = WarningsAsErrors( + flags=( + pattern("eslint --max-warnings 0", rf"{RUNS_ESLINT}[^\n]*\s--max-warnings[=\s]+0\b"), + pattern("biome ci", r"\bbiome\s+ci\b|\bbiome\b[^\n]*\s--error-on-warnings\b"), + # deno lint fails on any finding. + pattern("deno lint", RUNS_DENO_LINT), + ), + texts=_run_texts, + searched="CI configuration or the package scripts it runs", + hint="run eslint with --max-warnings 0 (or biome ci, or deno lint), so that a lint warning fails CI", +) + + +def _security_texts(context: Context) -> tuple[str, ...]: + return (*_run_texts(context), *existing_texts(context, (PACKAGE_JSON, *ESLINT_CONFIG_FILES))) + + +SECURITY_LINTS = SecurityLints( + texts=_security_texts, + lints=( + pattern("eslint-plugin-security", r"\beslint-plugin-security\b|@microsoft/eslint-plugin-sdl\b"), + pattern( + "semgrep JavaScript rules", + r"\bsemgrep\b[^\n]*--config[=\s]+[\"']?(?:p/)?(?:javascript|typescript|nodejs|react|expressjs)\b", + ), + ), +) + +DEPENDENCY_AUDIT = DependencyAudit( + tools=( + # npm audit signatures checks provenance, not known vulnerabilities. + pattern("npm audit", r"\bnpm\s+audit\b(?!\s+signatures)"), + pattern("pnpm audit", r"\bpnpm\s+audit\b"), + pattern("yarn audit", r"\byarn\s+(?:npm\s+)?audit\b"), + pattern("bun audit", r"\bbun\s+audit\b"), + *MULTI_ECOSYSTEM_AUDITS, + ), + hint="consider running npm audit (or pnpm audit, yarn npm audit or bun audit) in CI", +) + +# The rule's options are "off", 0, or anything else (a severity, or a severity and its limit). +MAX_LINES_PER_FUNCTION = re.compile( + r"""["']?max-lines-per-function["']?\s*:(?!\s*\[?\s*(?:["']off["']|0\b))""", re.IGNORECASE +) + + +def _max_lines_per_function(context: Context) -> Result: + """eslint's max-lines-per-function rule limits function length, when it is enabled and eslint runs.""" + config = first_existing(context, ESLINT_CONFIG_FILES) + if config is None or not MAX_LINES_PER_FUNCTION.search(context.project.read_text(config)): + return Result.failed( + "function length is never limited; enable eslint's max-lines-per-function rule " + f"or run {SUGGESTED_LIZARD} in CI" + ) + if not matching_names(_run_texts(context), LINTERS[:1]): + return Result.warned(f"max-lines-per-function is enabled in {config} but CI does not run eslint") + return Result.passed(f"function length is limited with eslint max-lines-per-function ({config})") + + +BIOME_CONFIG = configuration(("biome.json", "biome.jsonc")) +DENO_CONFIG = configuration(DENO_CONFIG_FILES) + +STATIC_ANALYSIS = StaticAnalysis( + analyzers=( + Analyzer(name="eslint", runs=re.compile(RUNS_ESLINT), configured=configuration(ESLINT_CONFIG_FILES)), + Analyzer(name="biome lint", runs=re.compile(RUNS_BIOME_LINT), configured=BIOME_CONFIG), + Analyzer(name="deno lint", runs=re.compile(RUNS_DENO_LINT), configured=DENO_CONFIG), + ), + texts=_run_texts, + hint="consider running eslint (or biome lint, or deno lint) in CI", +) + +TSCONFIG_FILES = ("tsconfig.json", "jsconfig.json") +# @tsconfig/strictest turns on strict, and more. +STRICTEST_BASE = re.compile(r"@tsconfig/strictest\b") + + +def _strict_type_checking(context: Context) -> str | None: + """Where strict type checking is enabled: a tsconfig.json, or Deno, which checks strictly unless told not to.""" + for path in TSCONFIG_FILES: + if json_value(context, path, "compilerOptions", "strict", comments=True) is True: + return f'"strict": true in {path}' + extends = json_value(context, path, "extends", comments=True) + bases = extends if isinstance(extends, list) else [extends] + if any(isinstance(base, str) and STRICTEST_BASE.search(base) for base in bases): + return f"@tsconfig/strictest in {path}" + deno = first_existing(context, DENO_CONFIG_FILES) + if deno is not None and json_value(context, deno, "compilerOptions", "strict", comments=True) is not False: + return f"Deno's default strict mode ({deno})" + return None + + +TYPE_CHECKING = TypeChecking( + checkers=( + Analyzer(name="tsc", runs=re.compile(r"(? str | None: + if any(json_value(context, PACKAGE_JSON, field) for field in DEPENDENCY_FIELDS): + return f"declared in {PACKAGE_JSON}" + deno = next((path for path in DENO_CONFIG_FILES if json_value(context, path, "imports", comments=True)), None) + return f"declared in {deno}" if deno else None + + +def _lockfile(context: Context) -> str | None: + return next((path for path in LOCKFILES if context.committed(path)), None) + + +# A package's lockfile is not published, so even a library commits one: its CI and contributors then install the +# dependencies it was tested with. +DEPENDENCIES = DependencyManagement( + managed=_declared_dependencies, + locked=_lockfile, + lock_hint="commit the package manager's lockfile (package-lock.json, pnpm-lock.yaml, yarn.lock, bun.lock or " + "deno.lock) and install from it in CI (npm ci, pnpm install --frozen-lockfile, yarn install --immutable)", +) + + +def _package_license(context: Context) -> str | None: + license_ = json_value(context, PACKAGE_JSON, "license") + return license_ if isinstance(license_, str) else None + + +LICENSE = DeclaredLicense(manifest=PACKAGE_JSON, declared=_package_license) + +NODE_MODULES = BuildOutputs( + name="node_modules", + globs=ignored_at_top_level("node_modules"), + paths=directories("node_modules"), + hint="consider ignoring node_modules/ (and coverage/ and dist/)", + # dist/ is left out: a GitHub Action written in JavaScript must commit its bundle. + artifacts=("node_modules/", "/coverage/", "*.tsbuildinfo"), +) + +_VERSION = r"(\d+(?:\.(?:\d+|x))*" +RUNTIME_VERSIONS = ( + re.compile(rf"\bnode-version\W+{_VERSION}|lts/[\w*-]+|latest|current|node)"), + re.compile(r"\bnode:(\d+(?:\.\d+)*)"), + re.compile(rf"\bdeno-version\W+v?{_VERSION}|canary|lts|rc)"), + re.compile(r"\bdenoland/deno:(?:\w+-)?(\d+(?:\.\d+)*)"), + re.compile(rf"\bbun-version\W+{_VERSION}|latest|canary)"), + re.compile(r"\boven/bun:(\d+(?:\.\d+)*)"), +) + +JAVASCRIPT_AND_TYPESCRIPT = Ecosystem( + label="JavaScript/TypeScript", + languages=frozenset({JAVASCRIPT, TYPESCRIPT}), + build_systems=(NPM, DENO), + source_suffixes={ + **dict.fromkeys(JS_SUFFIXES, JAVASCRIPT), + **dict.fromkeys(TS_SUFFIXES, TYPESCRIPT), + }, + toolchain=ToolchainPatterns( + language="JavaScript runtime", + versions=RUNTIME_VERSIONS, + # Not npx alone: other ecosystems' CI runs tools such as jscpd with it. + usage=re.compile( + r"\bnpm\s+(?:ci|install|i|test|t|run)\b|\b(?:pnpm|yarn)\s+\w|\bnode\s+(?:-\S|[\w./-]+\.[mc]?[jt]s\b)|\bsetup-(?:deno|bun)\b" + r"|\bdeno\s+(?:test|task|run|check|lint|fmt)\b|\bbun\s+(?:install|test|run)\b" + ), + ), + rules=( + versions_axis("runtime version", "JavaScript runtime"), + FORMATTERS, + COVERAGE, + TESTS, + WARNINGS, + WARNINGS_AS_ERRORS, + SECURITY_LINTS, + DEPENDENCY_AUDIT, + FunctionLimits(without_lizard=_max_lines_per_function), + STATIC_ANALYSIS, + TYPE_CHECKING, + DEPENDENCIES, + NODE_MODULES, + LICENSE, + ), +) diff --git a/project_validator/ecosystems/jvm.py b/project_validator/ecosystems/jvm.py new file mode 100644 index 0000000..5058e4a --- /dev/null +++ b/project_validator/ecosystems/jvm.py @@ -0,0 +1,381 @@ +"""Java and Kotlin on the JVM: built with Maven or Gradle, judged by the JDK versions CI runs. + +Maven and Gradle projects wire their tools into the build: a Maven plugin binds a goal to a lifecycle phase +(`check` runs in `mvn verify`), and a Gradle plugin adds its task to `check`. So the build files count as +CI configuration when CI runs the build far enough to run them. +""" + +import re +from collections.abc import Callable +from pathlib import Path, PurePosixPath + +from project_validator.core import Context, Result +from project_validator.ecosystems.model import BuildSystem, Ecosystem, ToolchainPatterns, versions_axis +from project_validator.evidence import ci_and_hook_texts, configuration, first_existing +from project_validator.rules import ( + MULTI_ECOSYSTEM_AUDITS, + SUGGESTED_LIZARD, + Analyzer, + BuildOutputs, + CoverageTools, + DependencyAudit, + DependencyManagement, + Formatter, + FunctionLimits, + SecurityLints, + StaticAnalysis, + TestSuite, + WarningPolicy, + WarningsAsErrors, + directories, + ignored_at_top_level, +) +from project_validator.text import Pattern, matching_names, pattern + +JAVA = "Java" +KOTLIN = "Kotlin" +POM = "pom.xml" +GRADLE_BUILD_FILES = ("build.gradle", "build.gradle.kts", "settings.gradle", "settings.gradle.kts") + + +def _output_beside(name: str, build_files: tuple[str, ...]) -> Callable[[Path], bool]: + def is_build_tree(directory: Path) -> bool: + return directory.name == name and any((directory.parent / build_file).is_file() for build_file in build_files) + + return is_build_tree + + +def _build_outputs(name: str, directory: str, *more: str) -> BuildOutputs: + return BuildOutputs( + name=f"the {name} directory", + globs=ignored_at_top_level(directory), + paths=directories(directory), + hint=f"consider ignoring {' and '.join((f'{directory}/', *more))}", + artifacts=(f"{directory}/", *more, "*.class"), + ) + + +MAVEN = BuildSystem( + name="Maven", + files=(POM,), + is_build_tree=_output_beside("target", (POM,)), + rules=(_build_outputs("target", "target"),), +) +GRADLE = BuildSystem( + name="Gradle", + files=GRADLE_BUILD_FILES, + is_build_tree=_output_beside("build", GRADLE_BUILD_FILES), + rules=(_build_outputs("build", "build", ".gradle/"),), +) + +# How far CI runs the build: a phase or task that compiles, and one that also runs the verification plugins. +MAVEN_COMMAND = r"(? tuple[str, ...]: + names = {POM, *GRADLE_BUILD_FILES} + return tuple(path for path in context.project.files() if PurePosixPath(path).name in names) + + +def _texts_when(runs: re.Pattern[str]) -> Callable[[Context], tuple[str, ...]]: + """CI configuration and pre-commit hooks, with the build files when CI runs the build as far as `runs`.""" + + def texts(context: Context) -> tuple[str, ...]: + found = ci_and_hook_texts(context) + if not any(runs.search(text) for text in found): + return found + return (*found, *(context.project.read_text(path) for path in _build_files(context))) + + return texts + + +_build_texts = _texts_when(RUNS_BUILD) +_verification_texts = _texts_when(RUNS_VERIFICATION) + + +def _bound_goal(artifact: str, goals: str) -> str: + """A Maven plugin whose execution binds one of `goals`, so that the lifecycle runs it.""" + return rf"{artifact}(?:(?!).)*?(?:{goals})" + + +def _setting(name: str, regex: str) -> Pattern: + return Pattern(name, re.compile(regex, re.IGNORECASE | re.DOTALL)) + + +GRADLE_PLUGIN = r"""(?:id\s*\(?\s*|apply\s+plugin:\s*|plugins\.apply\s*\(\s*)["']{}["']""" + + +def _gradle_plugin(plugin_id: str) -> str: + return GRADLE_PLUGIN.format(re.escape(plugin_id)) + + +def _spotless_config(context: Context) -> str | None: + return next((path for path in _build_files(context) if "spotless" in context.project.read_text(path)), None) + + +# Every formatter here has a built-in style (Spotless applies one of them). +FORMATTERS = Formatter( + name="Spotless, google-java-format, ktfmt or ktlint", + config_file=_spotless_config, + runs=( + _setting( + "Spotless", + rf"\bspotless:check\b|\bspotlessCheck\b|{_bound_goal('spotless-maven-plugin', 'check')}" + rf"|{_gradle_plugin('com.diffplug.spotless')}", + ), + _setting("fmt-maven-plugin", _bound_goal("fmt-maven-plugin", "check")), + _setting("google-java-format", r"\bgoogle-java-format\b[^\n]*--set-exit-if-changed\b"), + _setting("ktfmt", r"\bktfmt\b[^\n]*--set-exit-if-changed\b"), + _setting( + "ktlint", + rf"(? Result: + tests = (path for path in context.project.files() if TEST_SOURCES.search(path)) + count = sum(1 for path in tests if TEST_CASE.search(context.project.read_text(path))) + if count == 0: + return Result.failed( + "no JVM tests found (@Test methods or Kotest specs under src/test/java or src/test/kotlin)" + ) + return Result.passed(f"JVM tests found in {count} file(s)") + + +TESTS = TestSuite(found=_jvm_tests, runs=RUNS_TESTS, runner="mvn test or gradle test") + +# JaCoCo and Kover count branches; neither reports path coverage. +COVERAGE = CoverageTools( + searched="CI configuration, or the Maven or Gradle build CI runs", + texts=_verification_texts, + branch=( + _setting( + "JaCoCo", rf"\bjacoco:report\b|\bjacocoTestReport\b|{_bound_goal('jacoco-maven-plugin', 'report[\\w-]*')}" + ), + _setting( + "Kover", + r"\bkover\w*Report\b|\borg\.jetbrains\.kotlinx\.kover\b|kover-maven-plugin", + ), + ), +) + +XLINT = _setting("-Xlint:all", r"-Xlint(?::all)?(?![:\w-])") +# The Kotlin compiler warns by default; a Kotlin-only project is judged by warnings-as-errors. +KOTLIN_WARNINGS = _setting( + "Kotlin's default warnings", r"\bkotlin-maven-plugin\b|\borg\.jetbrains\.kotlin\b|\bkotlin\s*\(\s*[\"']jvm[\"']" +) + + +def _required_warnings(context: Context) -> tuple[Pattern, ...]: + return (XLINT,) if JAVA in context.structure.languages else (KOTLIN_WARNINGS,) + + +WARNINGS = WarningPolicy( + kind="JVM compiler warnings", + recognised=(XLINT, KOTLIN_WARNINGS), + required=_required_warnings, + texts=_build_texts, + separator=", ", +) + +WARNINGS_AS_ERRORS = WarningsAsErrors( + flags=( + _setting("-Werror", r"-Werror\b"), + _setting("allWarningsAsErrors", r"\ballWarningsAsErrors\b(?:\s*(?:=|\.set\s*\()\s*|>\s*)true\b"), + ), + texts=_build_texts, + searched="the Maven or Gradle build, or CI configuration", + hint="add -Werror to javac's compilerArgs (Kotlin: allWarningsAsErrors = true)", +) + +SECURITY_LINTS = SecurityLints( + texts=_verification_texts, + lints=( + _setting("Find Security Bugs", r"\bfindsecbugs-plugin\b|\bcom\.h3xstream\.findsecbugs\b"), + _setting("PMD security rules", r"\bcategory/java/security\.xml\b"), + ), +) + +DEPENDENCY_AUDIT = DependencyAudit( + tools=( + _setting( + "OWASP dependency-check", + rf"\bdependency-check:(?:check|aggregate)\b|\bdependencyCheck(?:Analyze|Aggregate)\b" + rf"|\bdependency-check\.sh\b|\bdependency-check/Dependency-Check_Action\b" + rf"|{_bound_goal('dependency-check-maven', 'check|aggregate')}", + ), + *MULTI_ECOSYSTEM_AUDITS, + ), + hint="consider OWASP dependency-check in CI (dependency-check-maven bound to check, or gradle " + "dependencyCheckAnalyze)", + texts=_verification_texts, +) + + +def _checkstyle_configs(context: Context) -> list[str]: + return [ + path + for path in context.project.files() + if "checkstyle" in PurePosixPath(path).name.lower() and path.endswith(".xml") + ] + + +DETEKT_CONFIG_FILES = ("detekt.yml", "config/detekt/detekt.yml", "detekt-config.yml") +METHOD_LENGTH = re.compile(r"""name\s*=\s*["']MethodLength["']""") +# detekt's LongMethod rule is active by default. +LONG_METHOD_OFF = re.compile(r"^\s*LongMethod:\s*\n\s*active:\s*false\b", re.MULTILINE) +RUNS_CHECKSTYLE = pattern("Checkstyle", r"\bcheckstyle\b") +RUNS_DETEKT = pattern("detekt", r"\bdetekt\b") + + +def _method_length_check(context: Context) -> str | None: + """The Checkstyle configuration that enables the MethodLength check, if any.""" + configs = _checkstyle_configs(context) + return next((path for path in configs if METHOD_LENGTH.search(context.project.read_text(path))), None) + + +def _long_method_active(context: Context) -> bool: + config = first_existing(context, DETEKT_CONFIG_FILES) + return config is None or not LONG_METHOD_OFF.search(context.project.read_text(config)) + + +def _method_length_enforced(context: Context) -> Result: + """Checkstyle's MethodLength check, or detekt's LongMethod rule, limits method length when the tool runs.""" + texts = _verification_texts(context) + checkstyle = _method_length_check(context) + if checkstyle and matching_names(texts, (RUNS_CHECKSTYLE,)): + return Result.passed(f"function length is limited with Checkstyle MethodLength ({checkstyle})") + if _long_method_active(context) and matching_names(texts, (RUNS_DETEKT,)): + return Result.passed("function length is limited with detekt LongMethod") + if checkstyle: + return Result.warned(f"MethodLength is enabled in {checkstyle} but Checkstyle does not run") + return Result.failed( + "function length is never limited; enable Checkstyle's MethodLength check, run detekt, " + f"or run {SUGGESTED_LIZARD} in CI" + ) + + +def _checkstyle_config(context: Context) -> str | None: + return next(iter(_checkstyle_configs(context)), None) + + +STATIC_ANALYSIS = StaticAnalysis( + analyzers=( + Analyzer(name="Error Prone", runs=re.compile(r"\berror_?prone\b|\bnet\.ltgt\.errorprone\b", re.IGNORECASE)), + Analyzer(name="SpotBugs", runs=re.compile(r"\bspotbugs\b", re.IGNORECASE)), + Analyzer( + name="PMD", + runs=re.compile( + rf"maven-pmd-plugin|\bpmd:check\b|\bpmd\s+check\b|{_gradle_plugin('pmd')}" + ), + ), + Analyzer(name="Checkstyle", runs=RUNS_CHECKSTYLE.regex, configured=_checkstyle_config), + Analyzer(name="detekt", runs=RUNS_DETEKT.regex, configured=configuration(DETEKT_CONFIG_FILES)), + ), + texts=_verification_texts, + hint="consider Error Prone, SpotBugs, PMD or Checkstyle (detekt for Kotlin), run by mvn verify or gradle check", +) + +MAVEN_DEPENDENCY = re.compile(r"") +GRADLE_DEPENDENCY = re.compile(r"^\s*(?:implementation|api|compileOnly|runtimeOnly|testImplementation)\b", re.MULTILINE) +GRADLE_LOCKS = ("gradle/verification-metadata.xml", "gradle.lockfile") +MAVEN_CONFIG = ".mvn/maven.config" +STRICT_CHECKSUMS = re.compile(rf"{MVN}(?:--strict-checksums|\s-C\b)") + + +def _managed(context: Context) -> str | None: + for path in _build_files(context): + text = context.project.read_text(path) + if PurePosixPath(path).name == POM and MAVEN_DEPENDENCY.search(text): + return "managed with Maven" + if PurePosixPath(path).name != POM and GRADLE_DEPENDENCY.search(text): + return "managed with Gradle" + return None + + +def _verified(context: Context) -> str | None: + """Gradle's dependency verification metadata or lockfile, or Maven failing on a bad checksum.""" + if lock := next((path for path in GRADLE_LOCKS if context.committed(path)), None): + return lock + if context.project.exists(MAVEN_CONFIG) and "--strict-checksums" in context.project.read_text(MAVEN_CONFIG): + return f"{MAVEN_CONFIG} (--strict-checksums)" + return "CI (mvn --strict-checksums)" if any(map(STRICT_CHECKSUMS.search, context.ci_texts)) else None + + +DEPENDENCIES = DependencyManagement( + managed=_managed, + locked=_verified, + lock_hint="verify them: commit gradle/verification-metadata.xml (gradle --write-verification-metadata sha256) " + "or a gradle.lockfile; with Maven, run with --strict-checksums", +) + +CLASS_FILES = BuildOutputs( + name="compiled classes", + globs=ignored_at_top_level(r"\*\.class"), + paths=("App.class",), + hint="consider ignoring *.class", + artifacts=("*.class",), +) + +_JDK = r"(\d+(?:\.\d+)*)" +JDK_VERSIONS = ( + re.compile(rf"\bjava-version\W+{_JDK}"), + re.compile(rf"\b(?:eclipse-temurin|openjdk|amazoncorretto|sapmachine|azul/zulu-openjdk\w*):{_JDK}"), + re.compile(rf"\bmaven:[\d.]+-[a-z-]+?-{_JDK}\b"), + re.compile(rf"\bgradle:[\w.-]*?jdk-?{_JDK}\b"), +) + +JVM = Ecosystem( + label="JVM", + languages=frozenset({JAVA, KOTLIN}), + build_systems=(MAVEN, GRADLE), + source_suffixes={".java": JAVA, ".kt": KOTLIN}, + # Kotlin and Groovy scripts are mostly Gradle builds and their helpers, which name no language. + ambiguous_suffixes={".kts": None, ".gradle": None}, + toolchain=ToolchainPatterns( + language="JDK", + versions=JDK_VERSIONS, + usage=re.compile(rf"{MAVEN_COMMAND}|{GRADLE_COMMAND}|\bsetup-java\b|\bjavac\b"), + ), + rules=( + versions_axis("JDK version", "JDK"), + FORMATTERS, + COVERAGE, + TESTS, + WARNINGS, + WARNINGS_AS_ERRORS, + SECURITY_LINTS, + DEPENDENCY_AUDIT, + FunctionLimits(without_lizard=_method_length_enforced), + STATIC_ANALYSIS, + DEPENDENCIES, + CLASS_FILES, + ), +) diff --git a/project_validator/ecosystems/php.py b/project_validator/ecosystems/php.py new file mode 100644 index 0000000..a8a9ebe --- /dev/null +++ b/project_validator/ecosystems/php.py @@ -0,0 +1,353 @@ +"""PHP: managed with Composer, judged by the PHP versions CI runs. + +PHP projects run their tools through Composer scripts (`composer test`, `composer cs-check`), so the scripts CI runs +count as CI configuration, as package scripts do for JavaScript. +""" + +import re +from collections.abc import Iterator +from pathlib import PurePosixPath + +from project_validator.core import Context, Result +from project_validator.ecosystems.model import BuildSystem, Ecosystem, ToolchainPatterns, versions_axis +from project_validator.evidence import ( + ci_and_hook_texts, + configuration, + existing_texts, + first_existing, + json_value, + scripts_run, +) +from project_validator.rules import ( + MULTI_ECOSYSTEM_AUDITS, + SUGGESTED_LIZARD, + Analyzer, + BuildOutputs, + CoverageTools, + DeclaredLicense, + DependencyAudit, + DependencyManagement, + Evidence, + Formatter, + FunctionLimits, + SecurityLints, + StaticAnalysis, + TestSuite, + TypeChecking, + WarningPolicy, + WarningsAsErrors, + directories, + ignored_at_top_level, +) +from project_validator.text import Pattern, pattern + +PHP_LANGUAGE = "PHP" +COMPOSER_JSON = "composer.json" +COMPOSER = BuildSystem(name="Composer", files=(COMPOSER_JSON,), languages=frozenset({PHP_LANGUAGE})) +PHPUNIT_CONFIG_FILES = ("phpunit.xml", "phpunit.xml.dist", "phpunit.dist.xml") + + +def _scripts(context: Context) -> dict[str, str]: + """The Composer scripts, by name; a script of several commands is one command per line.""" + scripts = json_value(context, COMPOSER_JSON, "scripts") + if not isinstance(scripts, dict): + return {} + commands = {name: [value] if isinstance(value, str) else value for name, value in scripts.items()} + return { + name: "\n".join(item for item in value if isinstance(item, str)) + for name, value in commands.items() + if isinstance(value, list) + } + + +# `composer test`, `composer run-script lint`, and `@lint` or `@composer lint` inside a script. +CALLS_SCRIPT = re.compile( + r"\bcomposer\s+(?:run(?:-script)?\s+)?(?P[\w:.-]+)|(?:^|\s)@(?!composer\b|php\b)(?P