README headings can cause training commands to be labeled as inference or evaluation
Source references: 6The classifier accepts the section-heading category before checking the entrypoint filename. For example, train.py under a “Demo” heading is initially labeled inference. The later structure check corrects it only when the script resolves locally, is at most 512 KiB, and reaches a pattern-score threshold; module entrypoints, unavailable scripts, or training implementations outside the limited patterns can remain misclassified.
If a downstream system uses this classification as a training-authorization or cost-control gate, a real training workload could be treated as lower-risk inference or evaluation, consuming substantial compute, changing model artifacts, or launching a long-running job.
The classification bypass is supported. `classify` returns the heading-derived category before checking entrypoint names, so `train.py` under “Demo” is initially labeled inference. Correction only examines a direct `.py` file that exists inside the repository and is at most 512 KiB, and it requires a score of at least four from a limited set of training patterns. Module entrypoints, missing/large scripts, or different training structures can remain misclassified. This could make a plan present training as lower-risk inference, although the skill itself does not execute it. Users can ask the author to prioritize explicit training entrypoints and mark unverifiable entrypoints as uncertain.
# authorization when both appear in the same title. if any(word in lowered for word in ["training", "train", "finetune", "fine-tune", "pretrain"]): return "training" if any(word in lowered for word in ["evaluation", "evaluate", "benchmark", "metrics", "validation"]) or re.search( r"\b(?:test|tests|testing)\b", lowered ): return "evaluation" if any(word in lowered for word in ["inference", "usage", "demo", "example", "text-to-image", "image-to-image", "transcribe"]): return "inference" return NoneShow 5 other places
section_category = infer_section_category(section) if section_category: return section_category for pattern, category in SCRIPT_CATEGORY_HINTS: if pattern.search(lowered): return categorydef referenced_python_script(command: str, readme_dir: Path) -> Optional[Path]: matched = PYTHON_ENTRYPOINT_RE.search(command) if not matched: return None root = readme_dir.resolve() candidate = (root / matched.group("path")).resolve() try: candidate.relative_to(root) except ValueError: return None if not candidate.is_file() or candidate.stat().st_size > 524_288: return None return candidate return [] evidence: List[str] = [] score = 0 for label, pattern, weight in TRAINING_STRUCTURE_SIGNALS: if pattern.search(content): evidence.append(label) score += weight return evidence if score >= 4 else []def infer_section_category(section: Optional[str]) -> Optional[str]: if not section: return None lowered = section.lower() # Training is the highest-risk interpretation. Check it before generic # headings such as "example" or "usage" so they cannot bypass training # authorization when both appear in the same title. if any(word in lowered for word in ["training", "train", "finetune", "fine-tune", "pretrain"]): return "training" if any(word in lowered for word in ["evaluation", "evaluate", "benchmark", "metrics", "validation"]) or re.search( r"\b(?:test|tests|testing)\b", lowered ): return "evaluation" if any(word in lowered for word in ["inference", "usage", "demo", "example", "text-to-image", "image-to-image", "transcribe"]): return "inference" return Nonedef training_structure_evidence(script: Path) -> List[str]: try: content = script.read_text(encoding="utf-8", errors="replace") except OSError: return [] evidence: List[str] = [] score = 0 for label, pattern, weight in TRAINING_STRUCTURE_SIGNALS: if pattern.search(content): evidence.append(label) score += weight return evidence if score >= 4 else []