Skip to content
skillsafebeta
← Report library
Purpose / Other

Caveman Optimize Skill Security Audit

What the author says it does (original text)

>

Independent security check

Security risks found

Files checked
1
Risks found
3
Could it run dangerous commands?Looks for programs run straight after downloading, remote control of your computer, and hidden commands.Risks found: 1
Medium risk

Approved evaluations execute fixture commands and repository checks that are not pre-bounded

Source references: 3
What we found

The Skill requires an “exact command” and runs the repository's focused code checks, but it does not restrict allowed commands, network access, or process privileges. An untrusted repository or test script could therefore execute arbitrary project code.

Why this matters

A malicious or compromised script could read credentials visible to the process, access the network, alter workspace files, or invoke local tools. Operator approval of the evaluation design reduces risk but does not itself establish that the commands are safe.

Legitimate use of this code

The commands are part of an operator-approved evaluation: the skill first requires a proposal naming the exact fixture and command, followed by approval before editing. Repository checks can execute project code, but the source does not direct hidden, arbitrary, or unapproved execution. The user should still confirm the commands and their network/process permissions when approving.

This assessment concerns the code and conditions shown, not proof that harm has occurred.
SKILL.md:67In the instructionsOpen original file
Propose one minimal candidate change and a **paired eval** before editing. Theevaluation must run baseline and candidate on identical fixed inputs and record:- the task-outcome or quality check that must remain acceptable;- the same token, byte, or provider-counted cost measure for both arms;- the exact fixture, command, and environment used; and- any confounder that prevents a fair comparison.
Show 2 other places
SKILL.md:82In the instructionsOpen original file
Keep the diff at the evidenced callsite and preserve existing safety controls.Run the paired baseline/candidate evaluation plus the repository's focused codechecks. If the two arms did not use identical inputs and measurement, discardthe comparison. If quality regresses or the resource result is inconclusive,
SKILL.md:75In the instructionsOpen original file
Ask for approval of the candidate and eval design. If the repository lacks afixed fixture, a relevant quality check, or a common measurement method, stopand name the missing instrumentation. Ordinary unit tests alone do not prove anoptimization.
Could it expose your files or keys?Looks for uploads of files containing passwords or keys, and keys written directly in the code.Risks found: 1
Medium risk

Recording the “exact environment” may carry secrets into the report

Source references: 2
What we found

The paired evaluation must record the “exact fixture, command, and environment,” and the final report includes commands and actual results. The text does not limit environment records to non-sensitive configuration or require redaction of tokens, keys, or secrets in command output.

Why this matters

If the agent copies complete environment variables or secret-bearing output into the report, sharing, committing, or uploading that report could disclose API keys, session tokens, local paths, or internal service details.

What this evidence establishes

“Record the exact environment” is ambiguous: it may mean ordinary run conditions, or could be misread as dumping all environment variables, which might expose tokens or keys. The text does not explicitly require reading or disclosing environment variables, and the final template only asks for check commands and results, so secret collection is not established. Users can ask the author to limit this to non-sensitive settings and redact output.

This assessment concerns the code and conditions shown, not proof that harm has occurred.
SKILL.md:70In the instructionsOpen original file
- the task-outcome or quality check that must remain acceptable;- the same token, byte, or provider-counted cost measure for both arms;- the exact fixture, command, and environment used; and- any confounder that prevents a fair comparison.
Show 1 other places
SKILL.md:99In the instructionsOpen original file
Recorded profile: <server observation, verbatim>Candidate: <file:line and approved change>Paired eval: <identical input/fixture, baseline result, candidate result>Quality check: <actual result>Code checks: <commands and actual results>Accounting: report-only profile; $0 opportunity band; no inferred or verified savings
Could it delete files or keep running?Looks for broad file deletion, disk overwrites, and programs set to start automatically.Risks found: 1
Low risk

The Skill makes lasting code changes and may automatically revert its candidate

Source references: 4
What we found

After two explicit approvals, the Skill edits a callsite and is instructed to revert that candidate edit when quality regresses or the resource result is inconclusive. It does not specify how to detect or preserve concurrent user changes made in the same area afterward.

Why this matters

This is normally an expected, controlled optimization workflow, but a mechanical revert could overwrite or conflict with later edits if the user or another tool modifies the same code concurrently.

Legitimate use of this code

The skill permits a persistent edit and rollback on failure, but only after explicit selection and approval, with the diff constrained to the evidenced callsite. Rollback is expressly limited to “only this candidate edit.” Concurrent-edit conflicts are a general version-control concern; the cited source does not instruct overwriting or discarding other changes. Users should still confirm workspace state and the rollback method before approval.

This assessment concerns the code and conditions shown, not proof that harm has occurred.
SKILL.md:75In the instructionsOpen original file
Ask for approval of the candidate and eval design. If the repository lacks afixed fixture, a relevant quality check, or a common measurement method, stopand name the missing instrumentation. Ordinary unit tests alone do not prove anoptimization.
Show 3 other places
SKILL.md:82In the instructionsOpen original file
Keep the diff at the evidenced callsite and preserve existing safety controls.Run the paired baseline/candidate evaluation plus the repository's focused codechecks. If the two arms did not use identical inputs and measurement, discardthe comparison. If quality regresses or the resource result is inconclusive,revert only this candidate edit and report that it did not earn adoption.
SKILL.md:53In the instructionsOpen original file
Present the available supported observations without ranking them. Include theid, the exact title, the exact observation, and `last_seen_at`. Ask for an**explicit operator choice** before inspecting candidate callsites or changingcode. If no supported current observation exists, stop with no edit.
SKILL.md:80In the instructionsOpen original file
## 4. Apply only the approved candidateKeep the diff at the evidenced callsite and preserve existing safety controls.Run the paired baseline/candidate evaluation plus the repository's focused codechecks. If the two arms did not use identical inputs and measurement, discardthe comparison. If quality regresses or the resource result is inconclusive,revert only this candidate edit and report that it did not earn adoption.
Could it bypass safety checks?Looks for skipped website security checks, excessive file access, or actions that skip your approval.No risks found
Could it mislead the AI or hide text?Checks the skill instructions for requests to ignore you, influence the report, or hide text in invisible characters.No risks found
Could it change links or payment recipients without asking?Looks for forced referral or payment changes combined with instructions to hide the change.No risks found

Inside this skill

5 instruction sections

The Skill first uses an authenticated Caveman CLI to read a report, accepts only four current report-only profiles, and explicitly excludes lifecycle data, retired IDs, and historic proposals.

View source
SKILL.md:18In the instructionsOpen original file
Require a logged-in Caveman CLI session and run:```bashcaveman opportunities list```Read only the `report_only_observations` array. Do not select from the lifecycle`data` array. Preserve each server-provided `title` and `observation` verbatim.Handle these exact repository-profile ids:
SKILL.md:40In the instructionsOpen original file
Never select or apply these retired ids:- `context-window-bloat`- `tool-catalog-utilization`- `verbose-tool-output`Treat any occurrence of a retired id in a stale proposal, local file, or oldresponse as historical context only. Never revive its money, recipe, orlifecycle claim. If the only actionable-looking item is `unlabeled-traffic`,

Before inspecting candidate callsites or changing code, the Skill requires an explicit operator choice; it then requires another approval for the specific candidate change and paired-evaluation design before editing.

View source
SKILL.md:53In the instructionsOpen original file
Present the available supported observations without ranking them. Include theid, the exact title, the exact observation, and `last_seen_at`. Ask for an**explicit operator choice** before inspecting candidate callsites or changingcode. If no supported current observation exists, stop with no edit.
SKILL.md:67In the instructionsOpen original file
Propose one minimal candidate change and a **paired eval** before editing. Theevaluation must run baseline and candidate on identical fixed inputs and record:- the task-outcome or quality check that must remain acceptable;- the same token, byte, or provider-counted cost measure for both arms;- the exact fixture, command, and environment used; and- any confounder that prevents a fair comparison.Ask for approval of the candidate and eval design. If the repository lacks afixed fixture, a relevant quality check, or a common measurement method, stop

After approval, the Skill modifies the evidenced callsite, runs baseline/candidate evaluation and code checks, and reverts the candidate edit if quality regresses or the result is inconclusive.

View source
SKILL.md:80In the instructionsOpen original file
## 4. Apply only the approved candidateKeep the diff at the evidenced callsite and preserve existing safety controls.Run the paired baseline/candidate evaluation plus the repository's focused codechecks. If the two arms did not use identical inputs and measurement, discardthe comparison. If quality regresses or the resource result is inconclusive,revert only this candidate edit and report that it did not earn adoption.

The Skill prohibits creating experiments, changing opportunity lifecycle state, or enabling an optimizer, and requires results to be described as fixture-specific observations rather than proof of production savings.

View source
SKILL.md:88In the instructionsOpen original file
Do not create a Caveman experiment or proposal, mark an opportunityimplemented, change its lifecycle, or switch on an optimizer. Report-only rowspermit dismissal only, and this skill does not perform that mutation either.
SKILL.md:107In the instructionsOpen original file
Never convert token or byte reduction into dollars without provider-complete,same-request accounting supplied by the product's verified methods. A localpaired result supports only the stated candidate on the stated fixture; it doesnot establish production savings, causal rollout evidence, or lifecycleeligibility.
Start here · InstructionsSKILL.md
caveman-optimize
Lines connect the instruction file to its sections, not an observed execution order. Select a section to read the source.
Files and check records1 files

Coverage and gaps

Content covered in each file

These are the source ranges included in this check, not a guarantee that every issue has been resolved.

  • SKILL.mdFull text included

This report is for the version above. We read the available code and instructions without running the skill or checking extra packages it installs. This is not a promise of safety: a different version or setup may behave differently.

  • SKILL.mdInstructions

Operations mentioned in code and instructions

Run commands
SKILL.md:20In the instructionsOpen original file
```bashcaveman opportunities list
Lines read
112
File checksum (to compare versions)
ca6eef52fb1a0f57a41e0d4c50cb3caa2ddd3e67aedbab9a8bad92dbb7554d04