Sitelet https://github.com/NVIDIA/SkillEvaluator/issues/43
Skip to content

policy: make metadata.author and metadata.id requirements profile-controlled #43

Description

@chrisknvidia

Context

OpenClaw reported that 0 of 8 skills passed the default external profile because metadata.author was missing.

The Agent Skills baseline requires name and description; metadata is optional. SkillEvaluator's current external profile nevertheless assigns HIGH severity to a missing author, making it blocking. At the same time, metadata.id is accepted as an extra metadata field but is not validated or used as catalog identity; local catalog entries are currently identified from content type plus relative path.

Registries may legitimately require stronger governance metadata, but those requirements should be explicit policy rather than universal schema assumptions.

Goal

Make author and stable-ID expectations profile-controlled: advisory for general public validation and optionally blocking for registries with a documented publication contract.

Requirements

  • Separate baseline Agent Skills schema requirements from registry publication policy.
  • Make missing metadata.author severity configurable, with a non-blocking external default.
  • Preserve profile control over malformed author shape and any domain restrictions.
  • Define an optional metadata.id contract, including syntax, length, normalization, and uniqueness scope.
  • Allow profiles to ignore, recommend, or require metadata.id.
  • Decide explicitly whether and how metadata.id participates in catalog identity, including fallback and migration behavior.
  • Surface the active policy and rationale clearly in reports and documentation.

Acceptance criteria

  • A skill containing valid name and description but no author is non-blocking under the default external profile.
  • A stricter registry profile can require author and/or metadata.id.
  • metadata.id, when present, receives bounded deterministic validation.
  • Duplicate or conflicting IDs are reported within the applicable registry/catalog scope.
  • Catalog identity behavior and migration compatibility are documented before IDs affect stored catalogs.
  • Profile, schema, catalog, reporting, and documentation tests cover advisory and blocking modes.

Activity

  1. AdemVessell commented on Aug 25, 2026

    @AdemVessell
    Contributor

    I did a source-pinned compatibility pass before coding and found that the author-policy mismatch is broader than the reported 0/8 missing-author case.

    At NVIDIA/skills commit 3a38625b, the 343 top-level skills/*/SKILL.md files break down as follows:

    • 100 have no metadata.author;
    • 243 have an author value, but only 92 match SkillEvaluator's current Name <email> shape;
    • 151 present authors fail that shape; common examples are team/organization attribution such as NVIDIA Corporation or NVIDIA cuOpt Team, which the current external profile makes blocking;
    • 0 declare metadata.id.

    Method: parse each top-level frontmatter mapping with yaml.safe_load, then apply the current _AUTHOR_SHAPE_PATTERN.fullmatch and count metadata.id presence. This means changing only SCHEMA.author_missing would still leave 151 current NVIDIA skills blocked on author shape.

    My proposed issue-sized contract is:

    1. Keep the existing severity mechanism for authors. Make both missing author and a present non-email team/organization author advisory under external; strict registry profiles can retain HIGH presence/shape findings and domain restrictions.
    2. Add identity.metadata_id_requirement = ignore | recommend | require. ignore controls absence only: every present ID still receives deterministic validation.
    3. Define metadata.id as an exact, unnormalized 1–64 character lowercase ASCII slug using ^[a-z0-9]+(?:-[a-z0-9]+)*$. It may differ from name; authenticated registry owner scope remains outside the self-declared skill document.
    4. Evaluate uniqueness only over an explicit discovered collection/catalog scope. Single-skill reports should say uniqueness was not evaluated, not claim uniqueness.
    5. Keep local catalog schema v1 path-keyed (content_type:relative_path) in this change. Validate declared IDs before catalog construction, but defer persistence/primary-identity migration until v2 or the update/delete and owner-scope contract in roadmap: scale Tier 2 overlap detection for registry-sized catalogs #41 is defined.
    6. Include the effective author severities, ID mode/contract, uniqueness scope, catalog identity mode, and policy digest in structured and human-readable reports.

    Three decisions would let me implement this without pre-empting registry policy:

    • Should the external default make present team/organization author forms advisory as well as missing authors?
    • Is the scoped, unnormalized 1–64 lowercase slug the intended metadata.id contract, with publisher qualification supplied by the registry?
    • Should policy: make metadata.author and metadata.id requirements profile-controlled #43 explicitly preserve the v1 path-based catalog and defer declared-ID persistence unless a v2 migration is accepted here?

    If those boundaries are acceptable, I can send a focused implementation with policy/schema/catalog-scope/report/docs tests. I have not opened a PR for this issue so the contract can be settled first.

  2. chrisknvidia commented on Aug 25, 2026

    @chrisknvidia
    CollaboratorAuthor

    @AdemVessell : Thanks for doing the source-pinned compatibility pass—this makes sense to me, and I agree the problem is broader than missing authors alone.
    My initial preference is:

    1. Under the external profile, both missing authors and legitimate team/organization attribution should be non-blocking. Strict registry profiles can continue to require the Name format and domain restrictions. We should still distinguish legitimate organization attribution from genuinely malformed or unsafe values.
    2. The proposed 1–64 character lowercase metadata.id slug and collection-scoped uniqueness model look reasonable.
    3. We should preserve the v1 path-based catalog identity in policy: make metadata.author and metadata.id requirements profile-controlled #43 and defer ID persistence and catalog migration until the registry identity/update contract is settled.

    I’d also like @rng1995 to review these policy boundaries before we finalize the contract and begin implementation.

    @rng1995 : could you please take a look and share your thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions