Skip to content
skillsafebeta
← Report library
Purpose / Other

Momentic Result Classification Skill Security Audit

What the author says it does (original text)

Classify or explain Momentic test run results using Momentic MCP tools.

Independent security check

Security risks found

Files checked
1
Risks found
3
Could it run dangerous commands?Looks for programs run straight after downloading, remote control of your computer, and hidden commands.No risks found
Could it expose your files or keys?Looks for uploads of files containing passwords or keys, and keys written directly in the code.Risks found: 1
Medium risk

Investigation can bring screenshots, DOM, network logs, and crash dumps into the analysis context and local workspace

Source references: 4
What we found

The Skill treats screenshots as the default evidence and can request full traces. Run assets may also contain DOM snapshots, HAR network entries, video, and browser crash dumps, while MCP places extracted results in a `.momentic-mcp` working directory.

Why this matters

If test pages expose tokens, personal information, customer data, or internal request content, that material may enter the model context and remain in local run artifacts, broadening access to sensitive data.

The investigation requests step screenshots and may request a potentially very large full trace; MCP also stores extracted run results under local `.momentic-mcp`. These materials can contain page content, DOM data, or sensitive information shown during testing, expanding local retention and model-processing exposure. HAR, video, and crash archives are documented as possible assets but are not required to be read by default. Users can request step-level minimization, full traces off by default, and a clear retention/deletion policy.

SKILL.md:25In the instructionsOpen original file
`momentic_get_step_result` — Returns the result of a specific step, with other information such as full step trace and before/after screenshots. Use `parentStepIdChain` for steps nested inside other steps. Only request `includeTrace=true` when you need it, because it can be very large.
Show 3 other places
SKILL.md:83In the instructionsOpen original file
    ├── metadata.json       attempt outcome and step results.    ├── console.json        optional browser console output.    └── assets/        ├── <snapshotId>.jpeg     before/after screenshot for each step (see attempt metadata.json for snapshot ID).        ├── <snapshotId>.html     before/after DOM snapshot for each step (see attempt metadata.json for snapshot ID).        ├── har-pages.log         HAR pages (ndjson).        ├── har-entries.log       HAR network entries (ndjson).        ├── resource-usage.ndjson CPU/memory samples taken during the attempt.        ├── <videoName>           video recording (when video recording is enabled).        └── browser-crash.zip     browser crash dump (only present on crash).```
SKILL.md:94In the instructionsOpen original file
When getting run results via the momentic MCP, tools such as `momentic_get_run` will return links to the MCP working directory (default `.momentic-mcp`). This directory will contain unzipped run result folders, following the structure above, named `run-result-<runId>`.
SKILL.md:55In the instructionsOpen original file
- Screenshots are the default truth source for page state. Use trace fields and DOM/HTML to explain why the screenshot changed or did not change.- Verify every causal claim. Do not say an overlay, side peek, modal, or menu was present unless the relevant before/after screenshot, URL, or DOM proves it.
Could it delete files or keep running?Looks for broad file deletion, disk overwrites, and programs set to start automatically.No risks found
Could it bypass safety checks?Looks for skipped website security checks, excessive file access, or actions that skip your approval.Risks found: 1
Medium risk

An explanation-only request may still trigger a persisted classification

Source references: 2
What we found

The Skill is advertised for explaining why a run failed, but its workflow also directs the agent to call the submission tool once investigation is complete. It does not require explicit user consent to write or distinguish read-only explanations from saved verdicts.

Why this matters

A user seeking analysis only could have the run's classification state changed in Momentic, potentially influencing later triage, reports, or human decisions.

The Skill explicitly covers users who only ask for an explanation, yet also provides a tool that persists the verdict and directs it to be called after investigation. It does not separate read-only answers from changes to the Momentic account or require confirmation. If the full workflow is applied to an explanation request, a run classification could be changed unexpectedly. Users can ask the author to make submission conditional on an explicit request to save.

SKILL.md:4In the instructionsOpen original file
description:  Classify or explain Momentic test run results using Momentic MCP tools.  Use when the user asks to categorize a failure, understand why a run failed,  triage test results, or compare run results to past run results.---
Show 1 other places
SKILL.md:29In the instructionsOpen original file
`momentic_submit_result_classification` — Persist your classification verdict for a run. Call this only after you have finished the investigation and are ready to record the final classification. Pass `runId` plus the fields described in the "Formal classification output" section below.
Could it mislead the AI or hide text?Checks the skill instructions for requests to ignore you, influence the report, or hide text in invisible characters.Risks found: 1
Medium risk

Untrusted text in test pages and run records can manipulate the classification workflow

Source references: 5
What we found

The Skill asks the agent to infer intent and causality from test names, descriptions, DOM, screenshots, and traces, but never says that this content is evidence only and must not be treated as agent instructions. Malicious text embedded in a page or test record could therefore be mistaken for operational direction.

Why this matters

An attacker-controlled page, test name, or log could steer the agent away from the investigation, cause unnecessary historical-run access, or produce and persist a false classification.

What this evidence establishes

The Skill does require the model to inspect test descriptions, screenshots, DOM/HTML, and traces to infer intent and causality, but the supplied text does not instruct it to execute commands found in those materials or show that page text controls tool calls. This leaves an undocumented trust-boundary gap, but the cited source is insufficient to establish that embedded text will be treated as agent instructions. Users can ask the author to declare all run content evidence-only and forbid it from triggering tool calls or persistence.

This assessment concerns the code and conditions shown, not proof that harm has occurred.
SKILL.md:27In the instructionsOpen original file
`momentic_get_test_steps_for_run` — Returns the simplified test steps recorded on a run (`stepsSnapshot`, `beforeStepsSnapshot`, `afterStepsSnapshot`). You can use this to understand the intent of the test if you need more information than what you can glean from the test name and description.
Show 4 other places
SKILL.md:55In the instructionsOpen original file
- Screenshots are the default truth source for page state. Use trace fields and DOM/HTML to explain why the screenshot changed or did not change.- Verify every causal claim. Do not say an overlay, side peek, modal, or menu was present unless the relevant before/after screenshot, URL, or DOM proves it.- Separate "the target is missing now" from "the browser is in the state where that target should exist." A missing target is often a symptom of an earlier failed action.
SKILL.md:175In the instructionsOpen original file
- Use test name, description, and, if needed, the simplified test steps returned by `momentic_get_test_steps_for_run` to determine what the test is intending to verify.- Failures outside that intent are unrelated, otherwise consider them related.- Any failures in setup (`beforeSteps` or `beforeResults`) or teardown (`afterSteps` or `afterResults`) are pretty much always considered unrelated.
SKILL.md:29In the instructionsOpen original file
`momentic_submit_result_classification` — Persist your classification verdict for a run. Call this only after you have finished the investigation and are ready to record the final classification. Pass `runId` plus the fields described in the "Formal classification output" section below.
SKILL.md:44In the instructionsOpen original file
- What is the test's intended behavior?- What is the earliest divergent step/container?- What did that step intend to do?- Which element/state did it actually interact with or observe?- What changed in the screenshot, URL, DOM, trace, or recovery log after the step?- Why is the later failure a consequence of that earlier divergence?
Could it change links or payment recipients without asking?Looks for forced referral or payment changes combined with instructions to hide the change.No risks found

Inside this skill

8 instruction sections

The Skill uses Momentic MCP to inspect the current failed run, its failing step, screenshots, and trace, then walks backward when the failing step already began from an incorrect state.

View source
SKILL.md:35In the instructionsOpen original file
1. Call `momentic_get_run` and identify the failing attempt, section (`beforeSteps`, main steps, or `afterSteps`), failing step, and any `parentStepIdChain`.2. Pull the failing step result with screenshots and trace. If the step is nested, also pull the nearest parent container or module result.3. Decide whether the failing step's before-screenshot is the correct baseline for that action. If it is already wrong, walk backward through the current run until you find the step/container that produced that bad state.4. For repeated modules or repeated workflows, compare invocations inside the same current run before comparing older runs. The later failure is often caused by an earlier invocation that succeeded, recovered, or left an invalid postcondition.

It can query historical runs of the same test for comparison and requires branch filtering when a branch name exists; history is intended only for specific questions after the current run is understood.

View source
SKILL.md:23In the instructionsOpen original file
`momentic_list_runs` — Recent runs for a test so you can compare the result of past runs over time. **Always pass `gitBranchName` when it exists on the run in question** so that it's more likely you're looking at the same version of the test. Omit it when you need runs from other branches. Pass `recovered=true` when you want to inspect recovered runs.
SKILL.md:130In the instructionsOpen original file
Past runs are comparison evidence, not a substitute for reconstructing the current run. Use them when the current run does not answer:- When did this test start failing?- What differed vs the last passing run?- Did the same action behave differently on an earlier run?- Is this a test weakness, an application change, a real application bug, or a temporary slowdown?

The final output contains one predefined failure category, recoverability, and confidence, and its explanation must identify the earliest divergence and broken postcondition.

View source
SKILL.md:202In the instructionsOpen original file
- When referencing past runs in final output, use clickable Momentic URLs rather than bare UUIDs: `https://app.momentic.ai/runs/<runId>`. Do not shorten UUIDs inside those URLs.- The reasoning must include the earliest divergent step/container and the broken postcondition it produced, not just the final failing step.```textReasoning: <a few sentences tied to the earliest divergence, screenshots/traces, past runs if used, and test intent>Category: <one id from the list>Recoverable: <RECOVERABLE | ONE_TIME_RECOVERABLE | NON_RECOVERABLE>Confidence: <high | medium | low>```
Start here · InstructionsSKILL.md
momentic-result-classification
Lines connect the instruction file to its sections, not an observed execution order. Select a section to read the source. 4 more sections are available in the original file.
Files and check records1 files

Coverage and gaps

Content covered in each file

These are the source ranges included in this check, not a guarantee that every issue has been resolved.

  • SKILL.mdFull text included

This report is for the version above. We read the available code and instructions without running the skill or checking extra packages it installs. This is not a promise of safety: a different version or setup may behave differently.

  • SKILL.mdInstructions

Operations mentioned in code and instructions

Read keys or account settings
SKILL.md:194In the instructionsOpen original file
- `NON_RECOVERABLE` — The failure cannot be automatically addressed and requires manual intervention.  - Examples: missing credentials; missing local files required for upload; outages likely caused by third-party systems where test steps cannot fix the issue.
SKILL.md:227In the instructionsOpen original file
- `TEST_SETUP` — Missing test data or files necessary to run the test, where the fix requires user action outside of the test itself.  - Examples: missing file for a file upload step; missing or incorrect credentials needed by the test.- `INFRA` — The failure was unrelated to the application or application code and was caused by an infrastructure outage, long load times, or some other issue due to outside factors.
Connect to websites
SKILL.md:201In the instructionsOpen original file
- Prefer human-readable references over UUIDs when the step/module can be identified colloquially: `module create-subpage-under-parent-page`, `the last invocation of module <name>`, `substep 4 (0-indexed)`, `the failed setup assertion`, etc …- When referencing past runs in final output, use clickable Momentic URLs rather than bare UUIDs: `https://app.momentic.ai/runs/<runId>`. Do not shorten UUIDs inside those URLs.- The reasoning must include the earliest divergent step/container and the broken postcondition it produced, not just the final failing step.
Lines read
233
File checksum (to compare versions)
2c298661f8e65271e4e0eb508cd49e765847675ef47f23f50d30871d8f3ac7cd