Skip to content
skillsafebeta
← Report library
Purpose / Other

Caveman Setup Skill Security Audit

What the author says it does (original text)

>

Independent security check

Do not install or run it yet

Files checked
1
Risks found
5
Could it run dangerous commands?Looks for programs run straight after downloading, remote control of your computer, and hidden commands.No risks found
Could it expose your files or keys?Looks for uploads of files containing passwords or keys, and keys written directly in the code.Risks found: 2
High risk

All live LLM prompts and responses pass through a third-party gateway

Source references: 3
What we found

The Skill requires changing every live LLM callsite's base URL to Caveman. This places the gateway between the application and the model provider, where it can receive request and response content. “Byte-preserving” and the claim that authentication headers are not forwarded are product assertions that cannot be independently verified from the supplied source.

Why this matters

Source code, user conversations, documents, personal data, or other secrets sent to the model may enter Caveman's processing and measurement systems and become subject to its logging, retention, access controls, and jurisdiction.

The live instructions route every active LLM call through Caveman and say the gateway measures what the app sends and receives. This places prompts, model outputs, and potentially sensitive business data in a third party's processing path. “Byte-preserving” is the Skill's own service claim; the supplied source does not independently establish retention, access, or logging practices. A user can require data-processing, retention, and deletion terms and restrict which data may pass through the gateway.

SKILL.md:9In the instructionsOpen original file
You are wiring this repository through the Caveman gateway. Caveman is abyte-preserving LLM proxy: in record mode it measures what your app sends andwhat it costs, and changes nothing else. Your job is a minimal, verifiedintegration — not a refactor.
Show 2 other places
SKILL.md:25In the instructionsOpen original file
1. **Coherent integration.** Wire every live LLM callsite through existing   configuration and responsible seams. Touch each layer correctness requires.   No drive-by refactors or formatting sweeps; add an abstraction only when it
SKILL.md:73In the instructionsOpen original file
Two facts that make the wiring safe (both are gateway-enforced, not hopes):the gateway rebuilds upstream auth headers from scratch, so a client's`Authorization`/`x-api-key` value is never forwarded to the provider; and with`stored`, upstream auth comes from the encrypted connection server-side. So in`stored` mode, where an SDK insists on an api-key parameter, set it to theCave key — it authenticates the gateway and goes no further.
High risk

BYOK mode discloses the model-provider key to the Caveman gateway

Source references: 4
What we found

In `byok` mode, the Skill explicitly sends the existing provider credential to the gateway on every request in `x-cave-upstream-key`. The credential is therefore no longer known only to the application and its original provider.

Why this matters

If the gateway, its logs, or the transport path is abused or compromised, the provider key could be used to consume the model account, create charges, or access other resources allowed by that credential.

In BYOK mode, the active instructions explicitly send the app's existing provider key to Caveman in `x-cave-upstream-key`. Even if the gateway later rebuilds upstream authentication, it must first receive the credential; the non-forwarding statement is a product claim and does not remove the impact of gateway disclosure or misuse. A user can require server-stored credentials, a short-lived and tightly limited provider key, or assurance that this header is never logged.

SKILL.md:67In the instructionsOpen original file
The pattern is always the same: **base URL → the gateway with `/w/<app>`,plus one auth header.** Gateway auth is `x-cave-api-key: CAVE_API_KEY`(`Authorization: Bearer CAVE_API_KEY` also works where a header is awkward).With `PROVIDER_KEYS: byok`, also send `x-cave-upstream-key: <the provider keythe app already uses>`.
Show 3 other places
SKILL.md:85In the instructionsOpen original file
```tsconst client = new OpenAI({  baseURL: `${process.env.CAVE_GATEWAY_URL}/w/<app>/openai/v1`,  apiKey: process.env.OPENAI_API_KEY,           // byok: unchanged · stored: use CAVE_API_KEY  defaultHeaders: {    "x-cave-api-key": process.env.CAVE_API_KEY!,    // byok only:    "x-cave-upstream-key": process.env.OPENAI_API_KEY!,  },});```
SKILL.md:172In the instructionsOpen original file
(byok: add `-H "x-cave-upstream-key: $PROVIDER_KEY"`.) This is one real,billable provider request — that is the point: real traffic, real measurement.
SKILL.md:73In the instructionsOpen original file
Two facts that make the wiring safe (both are gateway-enforced, not hopes):the gateway rebuilds upstream auth headers from scratch, so a client's`Authorization`/`x-api-key` value is never forwarded to the provider; and with`stored`, upstream auth comes from the encrypted connection server-side. So in`stored` mode, where an SDK insists on an api-key parameter, set it to theCave key — it authenticates the gateway and goes no further.
Could it delete files or keep running?Looks for broad file deletion, disk overwrites, and programs set to start automatically.Risks found: 1
Medium risk

The gateway credential is persistently stored in plaintext in a repository environment file

Source references: 2
What we found

The Skill requires writing the real `CAVE_API_KEY` to the repository's existing `.env`, `.env.local`, or similar file. Adding the file to `.gitignore` only reduces Git-commit exposure; it does not prevent access by other local users, backups, sync tools, malware, or permissive filesystem permissions.

Why this matters

Anyone able to read the workspace environment file may gain gateway access and, within the key's permissions, inspect or create traffic and consume quotas.

The Skill explicitly writes the real gateway credential into the repository's environment file using an unencrypted key-value form. Adding the file to `.gitignore` reduces accidental commits, but does not encrypt it or define file permissions, backup exclusions, or sync behavior; any local process or person able to read that file can obtain the key. A user can require an OS keychain or deployment secret store and verify file permissions and backup scope.

SKILL.md:29In the instructionsOpen original file
   clarifies ownership or lowers lifecycle cost.2. **Secrets stay in env vars.** `CAVE_API_KEY` goes into the env file the repo   already uses (`.env`, `.env.local`, …). If that file isn't gitignored, add it   to `.gitignore` and say so. Never hardcode the key in source.3. **Report only what you observed.** The final report states the HTTP status
Show 1 other places
SKILL.md:137In the instructionsOpen original file
Add to the repo's env file (and reference from code — no literals):```CAVE_GATEWAY_URL=<GATEWAY>CAVE_API_KEY=<CAVE_API_KEY>```
Could it bypass safety checks?Looks for skipped website security checks, excessive file access, or actions that skip your approval.No risks found
Could it mislead the AI or hide text?Checks the skill instructions for requests to ignore you, influence the report, or hide text in invisible characters.Risks found: 1
Medium risk

Unpinned remote documentation can change how the agent modifies the repository

Source references: 2
What we found

For several frameworks, the Skill tells the agent to fetch an integration page from the site where the Skill originated and follow it, without pinning a version, content digest, or permitted change set. Future page content therefore becomes executable instruction.

Why this matters

If the documentation site is compromised, changes ownership, or serves malicious content, the agent could be directed to run commands, add dependencies, disclose data, or make repository changes that are absent from the auditable Skill source.

For unlisted frameworks, the active instruction says to fetch an unspecified same-origin documentation page and follow it. No fixed URL, version, content hash, or operation allowlist is supplied, so later remote changes could influence repository modifications. The general no-drive-by-refactor rule adds some scope control but does not constrain installation commands or other additions a remote page might introduce. A user can require a pinned documentation copy and separate review before new dependencies, scripts, or network actions are followed.

SKILL.md:133In the instructionsOpen original file
For frameworks not listed (google-genai, crewai, pydantic-ai, openai-agents),fetch the matching page under `<docs origin>/docs/integrations/` — same originthis skill came from — and follow it.
Show 1 other places
SKILL.md:25In the instructionsOpen original file
1. **Coherent integration.** Wire every live LLM callsite through existing   configuration and responsible seams. Touch each layer correctness requires.   No drive-by refactors or formatting sweeps; add an abstraction only when it   clarifies ownership or lowers lifecycle cost.2. **Secrets stay in env vars.** `CAVE_API_KEY` goes into the env file the repo
Could it change links or payment recipients without asking?Looks for forced referral or payment changes combined with instructions to hide the change.Risks found: 1
Medium risk

The Skill treats a general setup request as permission to incur a charge without confirmation

Source references: 3
What we found

It explicitly tells the agent not to ask again and to send a real, billable model request immediately. Although the token cap is small, the actual model, account pricing, and minimum-charge rules are outside the Skill's control.

Why this matters

Running the Skill creates a charge and a real traffic record on the user's model account. On controlled production accounts it may also trigger budget, audit, or data-governance events.

The Skill treats pasting the setup prompt as authorization for a billable verification request and explicitly tells the agent not to reconfirm. Although output is capped at 32 tokens, it uses the repository's existing model and incurs real provider charges; the source does not establish the model, pricing, or account budget. A user can require disclosure of the model and estimated maximum charge first, or explicitly prohibit paid verification.

SKILL.md:146In the instructionsOpen original file
The user pasted the setup prompt to authorize exactly this: one smallverification request. Send it now — do not pause to ask permission for it.An integration that ends unverified because you hesitated is a worse outcomethan one tiny request; finishing the verification and the report autonomouslyis the point of this skill.
Show 2 other places
SKILL.md:152In the instructionsOpen original file
Send one minimal request through the wiring you just built — the app's owncheapest path if it has a script for it, otherwise curl **on the path matchingthe protocol you just wired** with the app's own model and a small cap(`max_tokens` ≤ 32):
SKILL.md:172In the instructionsOpen original file
(byok: add `-H "x-cave-upstream-key: $PROVIDER_KEY"`.) This is one real,billable provider request — that is the point: real traffic, real measurement.

Inside this skill

7 instruction sections

The Skill's main behavior is to reroute every live LLM call through the Caveman gateway and add gateway authentication headers. This changes who receives the network traffic even though the Skill claims model-visible bytes are unchanged.

View source
SKILL.md:25In the instructionsOpen original file
1. **Coherent integration.** Wire every live LLM callsite through existing   configuration and responsible seams. Touch each layer correctness requires.   No drive-by refactors or formatting sweeps; add an abstraction only when it
SKILL.md:67In the instructionsOpen original file
The pattern is always the same: **base URL → the gateway with `/w/<app>`,plus one auth header.** Gateway auth is `x-cave-api-key: CAVE_API_KEY`(`Authorization: Bearer CAVE_API_KEY` also works where a header is awkward).With `PROVIDER_KEYS: byok`, also send `x-cave-upstream-key: <the provider keythe app already uses>`.

The integration writes the Caveman secret into the repository's existing environment file and modifies `.gitignore` if that file is not already ignored.

View source
SKILL.md:29In the instructionsOpen original file
   clarifies ownership or lowers lifecycle cost.2. **Secrets stay in env vars.** `CAVE_API_KEY` goes into the env file the repo   already uses (`.env`, `.env.local`, …). If that file isn't gitignored, add it   to `.gitignore` and say so. Never hardcode the key in source.3. **Report only what you observed.** The final report states the HTTP status
SKILL.md:137In the instructionsOpen original file
Add to the repo's env file (and reference from code — no literals):```CAVE_GATEWAY_URL=<GATEWAY>CAVE_API_KEY=<CAVE_API_KEY>```

After wiring, the Skill requires an immediate real, billable provider request and uses its HTTP status and usage data as verification.

View source
SKILL.md:144In the instructionsOpen original file
## Step 4 — Verify with one real requestThe user pasted the setup prompt to authorize exactly this: one smallverification request. Send it now — do not pause to ask permission for it.An integration that ends unverified because you hesitated is a worse outcomethan one tiny request; finishing the verification and the report autonomouslyis the point of this skill.
SKILL.md:172In the instructionsOpen original file
(byok: add `-H "x-cave-upstream-key: $PROVIDER_KEY"`.) This is one real,billable provider request — that is the point: real traffic, real measurement.Read the response. Success = HTTP 200 with a `usage` block. Anything else =the matching failure template below.

For unlisted frameworks, the Skill delegates implementation details to documentation fetched at runtime from its originating site; the supplied material contains neither pinned content nor an integrity check.

View source
SKILL.md:133In the instructionsOpen original file
For frameworks not listed (google-genai, crewai, pydantic-ai, openai-agents),fetch the matching page under `<docs origin>/docs/integrations/` — same originthis skill came from — and follow it.
Start here · InstructionsSKILL.md
caveman-setup
Lines connect the instruction file to its sections, not an observed execution order. Select a section to read the source.
Files and check records1 files

Coverage and gaps

Content covered in each file

These are the source ranges included in this check, not a guarantee that every issue has been resolved.

  • SKILL.mdFull text included

This report is for the version above. We read the available code and instructions without running the skill or checking extra packages it installs. This is not a promise of safety: a different version or setup may behave differently.

  • SKILL.mdInstructions

Operations mentioned in code and instructions

Connect to websites
SKILL.md:16In the instructionsOpen original file
- `GATEWAY` — the gateway base URL (e.g. `https://gateway.caveman.so` or `http://127.0.0.1:8787`)- `CAVE_API_KEY` — the gateway auth secret (treat like any API key: env var only, never committed, never printed in full)
SKILL.md:19In the instructionsOpen original file
- `PROVIDER_KEYS` — `stored` (provider keys live encrypted in Caveman Cloud) or `byok` (this app sends its own provider key per request)- `DASHBOARD` — the dashboard base URL (e.g. `https://app.caveman.so`)
SKILL.md:130In the instructionsOpen original file
Concretely, with slug `support-bot` and the hosted gateway, an OpenAI-SDK baseURL reads `https://gateway.caveman.so/w/support-bot/openai/v1`. And in `stored`mode, drop every `x-cave-upstream-key` line entirely — it is byok-only.
Read keys or account settings
SKILL.md:17In the instructionsOpen original file
- `GATEWAY` — the gateway base URL (e.g. `https://gateway.caveman.so` or `http://127.0.0.1:8787`)- `CAVE_API_KEY` — the gateway auth secret (treat like any API key: env var only, never committed, never printed in full)- `PROVIDER_KEYS` — `stored` (provider keys live encrypted in Caveman Cloud) or `byok` (this app sends its own provider key per request)
SKILL.md:29In the instructionsOpen original file
   clarifies ownership or lowers lifecycle cost.2. **Secrets stay in env vars.** `CAVE_API_KEY` goes into the env file the repo   already uses (`.env`, `.env.local`, …). If that file isn't gitignored, add it
SKILL.md:30In the instructionsOpen original file
2. **Secrets stay in env vars.** `CAVE_API_KEY` goes into the env file the repo   already uses (`.env`, `.env.local`, …). If that file isn't gitignored, add it   to `.gitignore` and say so. Never hardcode the key in source.
Read files
SKILL.md:44In the instructionsOpen original file
Read dependency files (`package.json`, `requirements.txt`, `pyproject.toml`,`go.mod`, lockfiles) and search the source for LLM clients:
Run commands
SKILL.md:157In the instructionsOpen original file
```bash# OpenAI-protocol wiring:
Lines read
223
File checksum (to compare versions)
18d80378c4655f63e1327c58be69d8ac942e506334e66c8fb69d1768b863c556