Skip to content
Report library
Purpose / Other

Caveman Compress Skill Security Audit

What the author says it does (original text)

>

Independent security check

Do not install or run it yet

Files checked
10
Risks found
4
Could it run dangerous commands?Looks for programs run straight after downloading, remote control of your computer, and hidden commands.Risks found: 1
Medium risk

Without an API key, the Skill executes whichever claude program resolves from PATH

Source references: 4
What we found

The fallback locates an executable with shutil.which("claude") and runs it. The fixed argument list and absence of a shell reduce command injection, but the program's identity still depends on the current PATH.

Why this matters

If an untrusted project environment, installation, or compromised program places a same-named executable earlier in PATH, it runs with the Skill user's privileges and receives file content embedded in the compression prompt.

Supported, conditionally. Without ANTHROPIC_API_KEY, the skill resolves a program named claude from the current PATH and executes it. If another program or a low-trust directory has poisoned PATH, that executable receives the full prompt and runs with the process's permissions. The fixed argument list and absence of a shell reduce command-injection risk from file content, but executable identity is not verified. Users can restrict PATH, confirm the resolved Claude location, or ask the author to pin and verify a trusted executable.

scripts/compress.py:413In the codeOpen original file
    """    api_key = os.environ.get("ANTHROPIC_API_KEY")    if api_key:        try:            import anthropic            client = anthropic.Anthropic(api_key=api_key, timeout=CLAUDE_CALL_TIMEOUT_SECONDS)            msg = client.messages.create(                model=os.environ.get("CAVEMAN_MODEL", "claude-sonnet-4-5"),                max_tokens=8192,                messages=[{"role": "user", "content": prompt}],            )            # Tool-heavy models can put a tool_use or thinking block first; take            # the first text block instead of trusting content[0].            text = next((block.text for block in msg.content if getattr(block, "type", None) == "text"), "")            return strip_llm_wrapper(text.strip())        except ImportError:            pass  # anthropic not installed, fall back to CLI    # Fallback: use claude CLI (handles desktop auth).
Show 3 other places
scripts/compress.py:430In the codeOpen original file
            pass  # anthropic not installed, fall back to CLI    # Fallback: use claude CLI (handles desktop auth).    # Resolve binary via shutil.which so Windows .cmd/.bat shims (e.g.    # %APPDATA%\npm\claude.CMD) work without shell=True. On POSIX,    # shutil.which returns the same absolute path as the implicit lookup,    # so this is a no-op there. Falls back to bare "claude" if not found    # on PATH so subprocess raises a clear FileNotFoundError.    claude_bin = shutil.which("claude") or "claude"    try:        result = subprocess.run(            [                claude_bin,                "--print",                "--setting-sources",                "",                "--strict-mcp-config",            ],            input=prompt,            text=True,
scripts/compress.py:403In the codeOpen original file
    Prefers the Anthropic SDK when ANTHROPIC_API_KEY is set; otherwise falls    back to the ``claude --print`` CLI (which handles desktop auth).
scripts/compress.py:446In the codeOpen original file
            ],            input=prompt,            text=True,            capture_output=True,            check=True,            encoding="utf-8",            errors="replace",            timeout=CLAUDE_CALL_TIMEOUT_SECONDS,        )
Could it expose your files or keys?Looks for uploads of files containing passwords or keys, and keys written directly in the code.Risks found: 2
High risk

Target contents are sent to Anthropic; repair requests may include the complete original

Source references: 6
What we found

The initial request sends prose outside masked code blocks. If structural validation fails, the repair prompt embeds complete original_text, including code blocks and YAML frontmatter that were excluded initially. Sensitive-file checks examine names and path components, so tokens, customer data, internal instructions, or private code in ordinarily named files can still be transmitted.

Why this matters

Content crosses the local-machine boundary through the Anthropic SDK or the service used by Claude CLI. This may violate confidentiality, data-residency, or third-party-processing requirements.

Supported. The initial call sends the target body through the Anthropic SDK or Claude CLI. Code blocks are masked initially, but a validation failure builds a repair prompt containing the complete original_text, including YAML and code blocks. Refusal is based only on path/name patterns, so secrets, customer data, or private code in an ordinary-looking file can pass. Users should submit only files they may disclose to Anthropic and can ask for content scanning or explicit confirmation.

scripts/compress.py:403In the codeOpen original file
    Prefers the Anthropic SDK when ANTHROPIC_API_KEY is set; otherwise falls    back to the ``claude --print`` CLI (which handles desktop auth).    On Windows the CLI subprocess decoding defaults to the system codepage    (cp1251 / cp1252) and crashes on UTF-8 output — see issue #152. Pinning    ``encoding="utf-8"`` with ``errors="replace"`` matches the CLI's actual    native I/O and prevents the UnicodeDecodeError before validation can    report. Windows users with non-ASCII content can also set    ``ANTHROPIC_API_KEY`` to route through the SDK and skip the subprocess.    """    api_key = os.environ.get("ANTHROPIC_API_KEY")    if api_key:        try:            import anthropic            client = anthropic.Anthropic(api_key=api_key, timeout=CLAUDE_CALL_TIMEOUT_SECONDS)            msg = client.messages.create(                model=os.environ.get("CAVEMAN_MODEL", "claude-sonnet-4-5"),                max_tokens=8192,                messages=[{"role": "user", "content": prompt}],            )            # Tool-heavy models can put a tool_use or thinking block first; take
Show 5 other places
scripts/compress.py:601In the codeOpen original file
    # Refuse files that look like they contain secrets or PII. Compressing ships    # the raw bytes to the Anthropic API — a third-party boundary — so we fail    # loudly rather than silently exfiltrate credentials or keys. Override is    # intentional: the user must rename the file if the heuristic is wrong.    if is_sensitive_path(filepath):        raise ValueError(            f"Refusing to compress {filepath}: filename looks sensitive "            "(credentials, keys, secrets, or known private paths). "            "Compression sends file contents to the Anthropic API. "            "Rename the file if this is a false positive."        )
scripts/compress.py:503In the codeOpen original file
ORIGINAL (reference only):{original}COMPRESSED (fix this):{compressed}
scripts/compress.py:730In the codeOpen original file
        print("Fixing with Claude...")        fixed = call_claude(            build_fix_prompt(original_text, compressed, result.errors)        )
scripts/compress.py:413In the codeOpen original file
    """    api_key = os.environ.get("ANTHROPIC_API_KEY")    if api_key:        try:            import anthropic            client = anthropic.Anthropic(api_key=api_key, timeout=CLAUDE_CALL_TIMEOUT_SECONDS)            msg = client.messages.create(                model=os.environ.get("CAVEMAN_MODEL", "claude-sonnet-4-5"),                max_tokens=8192,                messages=[{"role": "user", "content": prompt}],            )            # Tool-heavy models can put a tool_use or thinking block first; take
scripts/compress.py:654In the codeOpen original file
    # Step 1: Compress (body only, frontmatter excluded)    print("Compressing with Claude...")    masked_body, code_blocks = mask_code_blocks(body)    masked_compressed = call_claude(build_compress_prompt(masked_body))    try:
Medium risk

A complete original copy persists outside the project directory

Source references: 4
What we found

The success path writes the original bytes under the XDG/user local-data directory. This copy does not automatically follow project deletion, archival, access-review, or secret-scanning workflows. SECURITY.md also says no files outside the provided path are accessed, which conflicts with the implemented backup and lock locations.

Why this matters

Old instructions, personal information, or internal material may remain after the project copy is cleaned and can still be collected by local backup, synchronization, or forensic tools. The contradictory description may cause users to overlook the copy.

Supported. A successful run writes the original bytes to an out-of-project user-data directory, with no expiry or automatic cleanup shown, so the copy can remain after project deletion, archival, or project-scoped scanning. The backup directory uses only the parent directory name and the file uses the stem, which can collide across separate projects with matching names and abort later runs. SECURITY.md's claim that no outside path is accessed also conflicts with implemented backup and lock files. Users can restrict data-directory permissions and ask for retention, cleanup, and stronger path isolation.

scripts/compress.py:93In the codeOpen original file
def _state_base_dir(kind: str) -> Path:    """Shared platform-aware base dir for caveman-compress state (backups, locks) — Windows uses %LOCALAPPDATA%, else $XDG_DATA_HOME or ~/.local/share."""    if _IS_WINDOWS:        local_appdata = os.environ.get("LOCALAPPDATA")        base = Path(local_appdata) if local_appdata else Path.home() / "AppData" / "Local"    else:        xdg = os.environ.get("XDG_DATA_HOME")        base = Path(xdg) if xdg else Path.home() / ".local" / "share"    return base / "caveman-compress" / kinddef backup_dir_for(filepath: Path) -> Path:    """Out-of-tree backup dir for filepath, keyed by its parent dir name — kept outside the source tree so skill auto-loaders don't re-ingest `.original.md` backups as live files."""    return _state_base_dir("backups") / filepath.parent.name
Show 3 other places
scripts/compress.py:691In the codeOpen original file
    # Save original as backup, then verify the backup readback before    # touching the input file. If the filesystem dropped bytes (encoding,    # antivirus, disk full), unlink the bad backup and abort instead of    # leaving the user with a corrupt backup + compressed primary.    backup_dir.mkdir(parents=True, exist_ok=True)    write_bytes_atomic(backup_path, original_raw)    if backup_path.read_bytes() != original_raw:        print(f"❌ Backup write verification failed: {backup_path}")
SECURITY.md:15In the instructionsOpen original file
- Does not execute user file content as code- Does not make network requests except to Anthropic's API (via SDK or CLI)- Does not access files outside the path the user provides- Does not use shell=True or string interpolation in subprocess calls- Does not collect or transmit any data beyond the file being compressed
scripts/compress.py:625In the codeOpen original file
    original_text, newline, original_raw = read_source(filepath)    # Store backup outside the source directory so skill auto-loaders don't    # re-ingest the `.original.md` copy as a live file. Mirror the source's    # parent-dir name + stem under a platform-aware base to reduce collisions.    backup_dir = backup_dir_for(filepath)    backup_path = backup_dir / (filepath.stem + ".original.md")
Could it delete files or keep running?Looks for broad file deletion, disk overwrites, and programs set to start automatically.No risks found
Could it bypass safety checks?Looks for skipped website security checks, excessive file access, or actions that skip your approval.No risks found
Could it mislead the AI or hide text?Checks the skill instructions for requests to ignore you, influence the report, or hide text in invisible characters.Risks found: 1
High risk

Passing structural validation can still silently change the meaning of persistent agent instructions

Source references: 5
What we found

Compression explicitly permits dropping phrasing, merging bullets, and removing examples. Validation checks headings, code blocks, URLs, some paths, bullet counts, and inline code, but not semantic equivalence, negation, priority, table contents, plain-text commands, numbers, or technical constraints. The target is then overwritten. Prompt-like text in the file can also influence the model's replacement.

Why this matters

Safety limits, deployment rules, tasks, or preferences may be weakened, reversed, or removed while the tool reports success. If an agent auto-loads the file, the change can affect later actions and decisions. A backup aids recovery but does not prevent altered instructions from being used first.

Supported. The skill permits merging bullets, dropping examples, and rewriting prose, while successful validation only checks headings, code blocks, URLs, paths, bullet counts, and inline code—not prose meaning, negation, priority, or table-cell content. Once those structural checks pass, model output overwrites the target. For persistent instructions such as CLAUDE.md, this can silently change later decisions. Source text is also embedded directly in the model prompt, so instruction-like content may steer the result. A backup enables recovery but does not prevent semantic drift.

SKILL.md:64In the instructionsOpen original file
### Compress- Use short synonyms: "big" not "extensive", "fix" not "implement a solution for", "use" not "utilize"- Fragments OK: "Run tests before commit" not "You should always run tests before committing"- Drop "you should", "make sure to", "remember to" — just state the action- Merge redundant bullets that say the same thing differently- Keep one example where multiple examples show the same pattern
Show 4 other places
scripts/validate.py:382In the codeOpen original file
def validate(original_path: Path, compressed_path: Path) -> ValidationResult:    result = ValidationResult()    orig = read_file(original_path)    comp = read_file(compressed_path)    validate_headings(orig, comp, result)    validate_code_blocks(orig, comp, result)    validate_urls(orig, comp, result)    validate_paths(orig, comp, result)    validate_bullets(orig, comp, result)    validate_inline_codes(orig, comp, result)    return result
scripts/compress.py:714In the codeOpen original file
        if result.is_valid:            print("Validation passed")            _write_target(filepath, compressed, backup_path, newline)            staging_path.unlink(missing_ok=True)            return True
README.md:162In the instructionsOpen original file
Caveman reduced counted tokens by about 46% on five listed fixtures. Validatorsconfirmed headings, code blocks, URLs, and file paths. They did not establishgeneral semantic or task-quality equivalence.
scripts/compress.py:464In the codeOpen original file
def build_compress_prompt(original: str) -> str:    return f"""Compress this markdown into caveman format.STRICT RULES:- Do NOT modify anything inside ``` code blocks- Do NOT modify anything inside a 4-space-indented code block either — those are code too, and they are validated- Do NOT modify anything inside inline backticks- Preserve ALL URLs exactly- Preserve ALL headings exactly- Preserve file paths and commands- Return ONLY the compressed markdown body — do NOT wrap the entire output in a ```markdown fence or any other fence. Inner code blocks from the original stay as-is; do not add a new outer fence around the whole file.Only compress natural language.TEXT:{original}"""
Could it change links or payment recipients without asking?Looks for forced referral or payment changes combined with instructions to hide the change.No risks found

Inside this skill

6 instruction sections

The Skill accepts only files classified as natural language. It rejects common code/config extensions, backup files, and paths whose names look credential-related. This protection is mainly extension- and path-name-based, not a scan of file contents.

View source
scripts/detect.py:119In the codeOpen original file
def should_compress(filepath: Path) -> bool:    """Return True if the file is natural language and should be compressed."""    if not filepath.is_file():        return False    # Skip backup files    if filepath.name.endswith(".original.md"):        return False    return detect_file_type(filepath) == "natural_language"
scripts/compress.py:197In the codeOpen original file
def is_sensitive_path(filepath: Path) -> bool:    """Heuristic denylist for files that must never be shipped to a third-party API."""    name = filepath.name    if SENSITIVE_BASENAME_REGEX.match(name):        return True    # Normalize every component, not only basename: directories named    # `api-keys`, `private_keys`, or singular `secret` are equally sensitive.    normalized_parts = {        re.sub(r"[_\-\s.]", "", part.lower()) for part in filepath.parts    }    if normalized_parts & SENSITIVE_PATH_COMPONENTS:        return True    return any(        token in part        for part in normalized_parts        for token in SENSITIVE_NAME_TOKENS    )

Code blocks are replaced with markers before the initial Claude compression request. If validation fails, the repair request includes both the complete original text and the compressed version.

View source
scripts/compress.py:654In the codeOpen original file
    # Step 1: Compress (body only, frontmatter excluded)    print("Compressing with Claude...")    masked_body, code_blocks = mask_code_blocks(body)    masked_compressed = call_claude(build_compress_prompt(masked_body))    try:        compressed_body = restore_code_blocks(masked_compressed, code_blocks)    except ValueError as error:
scripts/compress.py:503In the codeOpen original file
ORIGINAL (reference only):{original}COMPRESSED (fix this):{compressed}
scripts/compress.py:730In the codeOpen original file
        print("Fixing with Claude...")        fixed = call_claude(            build_fix_prompt(original_text, compressed, result.errors)        )

On success, the target is replaced in place with the compressed result, while its original bytes are stored as a persistent backup under the user's data directory. Writes use a temporary file and atomic replacement.

View source
scripts/compress.py:93In the codeOpen original file
def _state_base_dir(kind: str) -> Path:    """Shared platform-aware base dir for caveman-compress state (backups, locks) — Windows uses %LOCALAPPDATA%, else $XDG_DATA_HOME or ~/.local/share."""    if _IS_WINDOWS:        local_appdata = os.environ.get("LOCALAPPDATA")        base = Path(local_appdata) if local_appdata else Path.home() / "AppData" / "Local"    else:        xdg = os.environ.get("XDG_DATA_HOME")        base = Path(xdg) if xdg else Path.home() / ".local" / "share"    return base / "caveman-compress" / kinddef backup_dir_for(filepath: Path) -> Path:    """Out-of-tree backup dir for filepath, keyed by its parent dir name — kept outside the source tree so skill auto-loaders don't re-ingest `.original.md` backups as live files."""    return _state_base_dir("backups") / filepath.parent.name
scripts/compress.py:695In the codeOpen original file
    # leaving the user with a corrupt backup + compressed primary.    backup_dir.mkdir(parents=True, exist_ok=True)    write_bytes_atomic(backup_path, original_raw)    if backup_path.read_bytes() != original_raw:        print(f"❌ Backup write verification failed: {backup_path}")
scripts/compress.py:714In the codeOpen original file
        if result.is_valid:            print("Validation passed")            _write_target(filepath, compressed, backup_path, newline)            staging_path.unlink(missing_ok=True)            return True
Start here · InstructionsSKILL.md
caveman-compress
Lines connect the instruction file to its sections, not an observed execution order. Select a section to read the source.

File reference map

References: 2
Files making referencesReferenced content
Lines show actual file references, not execution order. Select a node to highlight its connections and inspect the files and source locations. Dashed lines include files that still need locating.
Files and check records10 files

Coverage and gaps

Content covered in each file

These are the source ranges included in this check, not a guarantee that every issue has been resolved.

  • SKILL.mdFull text included
  • scripts/__init__.pyFull text included
  • scripts/__main__.pyFull text included
  • scripts/benchmark.pyFull text included
  • scripts/cli.pyFull text included
  • scripts/compress.pyFull text included
  • scripts/detect.pyFull text included
  • scripts/validate.pyFull text included
  • SECURITY.mdFull text included
  • README.mdFull text included

This report is for the version above. We read the available code and instructions without running the skill or checking extra packages it installs. This is not a promise of safety: a different version or setup may behave differently.

  • README.mdSupporting file
  • SECURITY.mdSupporting file
  • SKILL.mdInstructions
  • scripts/__init__.pyScript
  • scripts/__main__.pyScript
  • scripts/benchmark.pyScript
  • scripts/cli.pyScript
  • scripts/compress.pyScript
  • scripts/detect.pyScript
  • scripts/validate.pyScript

Operations mentioned in code and instructions

Connect to websites
README.md:2In the instructionsOpen original file
<p align="center">  <img src="https://em-content.zobj.net/source/apple/391/rock_1faa8.png" width="80" /></p>
README.md:177In the instructionsOpen original file
This skill is part of the [caveman](https://github.com/JuliusBrussee/caveman) toolkit.
Read files
scripts/benchmark.py:26In the codeOpen original file
def benchmark_pair(orig_path: Path, comp_path: Path):    orig_text = orig_path.read_text(encoding="utf-8", errors="ignore")    comp_text = comp_path.read_text(encoding="utf-8", errors="ignore")
scripts/benchmark.py:27In the codeOpen original file
    orig_text = orig_path.read_text(encoding="utf-8", errors="ignore")    comp_text = comp_path.read_text(encoding="utf-8", errors="ignore")
scripts/compress.py:162In the codeOpen original file
        raise OSError(f"Refusing to open lock file through a symlink: {lock_path}")    fd = os.open(lock_path, os.O_CREAT | os.O_RDWR | _O_NOFOLLOW, 0o600)    try:
Run commands
scripts/compress.py:16In the codeOpen original file
import statimport subprocessimport sys
scripts/compress.py:223In the codeOpen original file
    STARTS and ENDS with a fence line. An ordinary README section —    ```bash npm install``` , prose, ```bash npm test``` — came back with its    first and last fence markers deleted and its two code blocks merged into
scripts/compress.py:406In the codeOpen original file
    On Windows the CLI subprocess decoding defaults to the system codepage    (cp1251 / cp1252) and crashes on UTF-8 output — see issue #152. Pinning
Install extra software packages
scripts/compress.py:223In the codeOpen original file
    STARTS and ENDS with a fence line. An ordinary README section —    ```bash npm install``` , prose, ```bash npm test``` — came back with its    first and last fence markers deleted and its two code blocks merged into
README.md:151In the instructionsOpen original file
- File paths (`/src/components/...`)- Commands (`npm install`, `git commit`)- Technical terms, library names, API names
SKILL.md:51In the instructionsOpen original file
- File paths (`/src/components/...`, `./config.yaml`)- Commands (`npm install`, `git commit`, `docker build`)- Technical terms (library names, API names, protocols, algorithms)
Read keys or account settings
scripts/compress.py:65In the codeOpen original file
# them ships raw bytes to the Anthropic API — a third-party data boundary that# developers on sensitive codebases cannot cross. detect.py already skips .env# by extension, but credentials.md / secrets.txt / ~/.aws/credentials would
scripts/compress.py:66In the codeOpen original file
# developers on sensitive codebases cannot cross. detect.py already skips .env# by extension, but credentials.md / secrets.txt / ~/.aws/credentials would# slip through the natural-language filter. This is a hard refuse before read.
scripts/compress.py:70In the codeOpen original file
    r"(?ix)^("    r"\.env(\..+)?"    r"|\.netrc"
Change files
scripts/compress.py:165In the codeOpen original file
        if os.fstat(fd).st_size == 0:            os.write(fd, b"\0")  # msvcrt.locking needs at least one byte in the file to lock        os.lseek(fd, 0, 0)
scripts/compress.py:254In the codeOpen original file
def write_text_atomic(path: Path, text: str, newline: str = "\n") -> None:    """Write ``text`` to ``path`` atomically as UTF-8.
scripts/compress.py:257In the codeOpen original file
    Path.write_text() truncates the destination before encoding the string —    a UnicodeEncodeError (or any other failure) partway through leaves a
Lines read
1,834
File checksum (to compare versions)
1a5de1df97be5922d1adcea52955e45164a60665ae89456add6621531c522ec4