XAVET

XAVET

A demo is not engineering evidence.

X-as-Code Assessment and Verification of Engineering Toolchains

Measure before we trust.

An independent assessment instrument for establishing what an AI-enabled AUTOSAR X-as-Code engineering toolchain has actually demonstrated — built for the assessors, quality and safety specialists and architects who have to answer for that decision. Under development; independently owned.

01 Position

The problem

AI engineering tools demo well.

We would never accept a physical component on supplier claims and a successful demo alone. Why should an AI-enabled engineering toolchain be different?

A successful demo shows

  • A compelling happy path.
  • A plausible AUTOSAR artifact.
  • Speed and usability worth exploring.

It does not establish

  • Repeatability from a clean baseline.
  • Independent deterministic acceptance.
  • Behaviour when inputs are invalid or a service is unavailable.
  • Evidence an outside reviewer can inspect.

Those are different questions, and they are the ones that decide whether an engineering organisation can rely on a tool. The people who have to answer them have no repeatable way to tell an engineering tool from a wrapper around a chat model: each evaluation is improvised, each conclusion rests on the confidence of whoever ran it, and nothing carries over to the next tool.

XAVET exists to make that judgement repeatable: same claim, same procedure, same evidence, same conclusion. The point is to reject unbounded claims — not innovation.

02 Procedure

The method

An assessment is a fixed, documented pipeline. Every step is versioned and repeatable, and nothing is accepted because it looked right.

  1. Freeze the claim.

    The developer of the system under test fills in a controlled information package: which engineering layers and operations the toolchain claims, in which formats, with which boundaries and stated limitations. Nothing is assessed that was not claimed.

  2. Derive the scope deterministically.

    Applicable criteria, gates, layers, operations and test cases are derived from that frozen profile against published catalogs — with no code written for a particular vendor. The same profile always produces the same plan; missing information produces a recorded finding, never a guess.

  3. Hand over a calibrated slice.

    The developer receives a slice of a trusted reference project: a real AUTOSAR-style project whose correct result is known in advance, shipped with deliberately broken variants that prove the check can actually detect a fault. Expected results are frozen before anything is executed.

  4. Run the developer's own workflow.

    Human operator-reviewers execute only the documented workflow of the system under test — no shortcuts supplied by its developer — and record Pass, Fail, Blocked or N/A at every control point.

  5. Ingest artifacts and evidence.

    Exported artifacts, reports, logs, configuration and hashes are collected for independent inspection, alongside the reviewers' procedural observations.

  6. Report at criterion level.

    Deterministic checks and reviewer findings produce an evidence-backed report: what was demonstrated, for which use case, under which conditions — and what was not.

A concrete AUTOSAR test, not a chat benchmark

The first complete vertical slice asks one question an AUTOSAR engineer can check: can the system add a defined interface without chat as the only path, and produce correct, inspectable ARXML?

Open baseline
Load the prepared reference project from its frozen initial state.
Apply change
Add a defined interface and bind it to the intended software component.
Validate
Invoke the documented validation path and retain the result.
Generate
Produce ARXML and collect logs, configuration, actions and hashes.
Compare
Normalise the export and compare its semantics against the frozen expected graph.

Because an assessment runs on a controlled reference project, an organisation never has to hand a real internal project to the developer of the system under test: its requirements, models, product data and tool configurations stay where they are. The reference is treated as a measuring device rather than an example — it passed a formal multi-round review trust gate, and the sensitivity of its oracle is itself measurable through mutation suites.

03 Mandatory gates

G0

Independent mandatory path — the flagship gate

Every mandatory engineering operation must be invokable with all generative-model services unavailable.

A structured editor, a versioned command line and a typed API are all acceptable paths. Chat may be an optional convenience; it may never be the only route to authoritative editing, validation, generation, build, test or export. An interface that can only be reached by asking an agent is not an engineering path.

G0 does not prohibit AI — it requires an independent path for the operations that must not fail. “Mandatory-AI” is not an accepted classification, and that is the whole point: G0 turns the difference between an engineering tool and a wrapper around a chat model into something an operator can test in an afternoon, with the network unplugged.

It is the first of six mandatory gates. Each one is a question asked before a claim is trusted.

G2 is the central assurance boundary: the same model that generated an artifact cannot be the acceptance authority for that artifact. A gate result applies to the affected use case and scope — it is never generalised to unrelated capabilities, and never averaged away.

04 Boundary

No LLM in the verdict path

System under test

AI may be in the system under test.

Verdict path

It is never in the decision.

The acceptance oracle, the mandatory gates and the report verdicts do not depend on a generative model. A model's statement that an artifact is correct is not an acceptance result; a deterministic, versioned check against a predeclared expected result is.

The same line holds on the assessor's side — assistance without authority. An optional assistant is confined to read, propose and run-deterministic capabilities and is explicitly denied decide and approve; its output stays a draft until a human accepts it through the same interface that works without it, and if the model provider is unreachable an assessment continues without assistance.

Prompt text is not an authorization boundary. Data scopes, available tools, frozen-state rules and human-only actions are enforced deterministically and written to an audit trail.

05 Result

Verdicts

Not a score. Not a rank. Not a leaderboard.

A result is a bounded, evidence-backed statement about what a system demonstrated, under which conditions, for which claimed scope. There is no global pass to win: one safe bounded use case is not approval of every capability.

Reject
The engineering-system claim is not supported for the affected scope.
Bounded pilot
Correctness demonstrated on approved scenarios, with scope restricted and a validated path that does not depend on a generative model retained alongside it.
Exploratory
Assessable, but higher requirements are unmet; the outputs are not engineering evidence.

Verdicts are recorded per criterion and per use case, and the headline states the worst case over the assessed set. Above Bounded pilot the methodology defines further states — controlled assistant use, then controlled authoritative use — each demanding stronger evidence than the one below. The useful effect is the same at every level: a vague yes-or-no argument about AI becomes a portfolio of bounded decisions, each with its own conditions and evidence.

06 State

Status

XAVET is pre-release and under active development. What follows is where the work actually stands.

Built
  • The trusted reference project, its deterministic self-validation and its deliberately broken variants.
  • The assessment definition package — vocabulary, criteria, applicability rules, schemas and the G0–G5 gate catalog — and the deterministic scope and test-plan generator built on it.
  • The assessor workspace with its content-addressed audit trail, its command-line interface and a frozen protocol for client tools, exercised by a second independent implementation before it was frozen.
  • Standalone checks on a candidate artifact: validation of an ARXML against an operator-supplied official schema, a tool-neutral reader that turns any ARXML into a normalised graph, and a comparator that sets that graph against the reference's expected one — producing findings, not verdicts.
In progress
  • The assessor workbench client over that protocol.
Not started
  • Mapping comparator findings to criteria and gates, and the assessment report.
  • Assessment-session and evidence-package workflows — the part that carries the standalone checks into a recorded assessment.
  • The optional assessor assistant, under the boundary described above.
  • Field validation against real systems under test.

Today XAVET can define and prepare a reproducible assessment, and check a candidate ARXML against the reference on its own. It cannot yet run an assessment end to end or turn findings into a verdict.

No assessment has been run yet, and the first pilot has not been selected. Whichever system it turns out to be, the first run measures the instrument as much as the system under test: it is there to calibrate the method, expose its gaps and show what the evidence is worth.

No third-party assessment has been performed. XAVET is not a compliance service — it does not certify or qualify any toolchain, and does not make one compliant with any standard. No license has been published and the project is not open source at this time; the instrument's sources are private. This page presents the project. It does not distribute it.

07 Contact

Author / contact

XAVET is built and owned independently by Konstantin Tarandevich. An instrument for measuring a developer’s claims cannot belong to that developer, so independence here is a property of the measurement rather than a courtesy.

The useful next step is not a standard or a rollout. It is one bounded pilot: one representative AI-enabled toolchain, one frozen scenario, one named operator and one independent reviewer — producing a measured result for that capability, a calibrated list of method and tooling gaps, and evidence for deciding what deserves the next investment.

If you are evaluating such a toolchain, building one, or want to argue with the methodology, either link reaches me.

The point is to stop comparing AI engineering systems by the quality of their demos, and start comparing them by the strength of their evidence. AI where it helps. Determinism where it matters.