# Architecture
Source: https://docs.shipsec.ai/architecture

System design and module responsibilities for ShipSec Studio

## What is ShipSec Studio?

ShipSec Studio is an open-source, no-code security workflow orchestration platform. Build, execute, and monitor security automation workflows through a visual interface — focus on security, not infrastructure.

***

## System Overview

```
┌────────────────────────────────────────────────────────────────────────┐
│                              FRONTEND                                  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │
│  │   Visual    │  │  Terminal   │  │  Timeline   │  │   Config    │    │
│  │   Builder   │  │   Viewer    │  │   Replay    │  │    Panel    │    │
│  │ (ReactFlow) │  │  (xterm.js) │  │  (Zustand)  │  │  (Forms)    │    │
│  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘    │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ REST API + Unified SSE
┌───────────────────────────────────▼─────────────────────────────────────┐
│                               BACKEND                                   │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐       │
│  │Workflows │ │ Secrets  │ │ Storage  │ │  Trace   │ │  Auth    │       │
│  │  + DSL   │ │(AES-256) │ │ (MinIO)  │ │ Events   │ │ (Clerk)  │       │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘       │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐       │
│  │ Webhooks │ │Schedules │ │ Agents   │ │Human     │ │Integr-   │       │
│  │          │ │ (CRON)   │ │          │ │Inputs    │ │ations    │       │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘       │
└───────────────────────────────────┬─────────────────────────────────────┘
                                    │ Temporal Client
┌───────────────────────────────────▼─────────────────────────────────────┐
│                              TEMPORAL                                   │
│            Workflow Orchestration • Retry Logic • Durability            │
└───────────────────────────────────┬─────────────────────────────────────┘
                                    │ Activity Execution
┌───────────────────────────────────▼─────────────────────────────────────┐
│                               WORKER                                    │
│  ┌─────────────────────────────────────────────────────────────────┐    │
│  │                    COMPONENT REGISTRY                           │    │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐            │    │
│  │  │  Tools   │ │    AI    │ │  Human   │ │   Core   │            │    │
│  │  │(Security)│ │  Agents  │ │ in Loop  │ │  Utils   │            │    │
│  │  └──────────┘ └──────────┘ └──────────┘ └──────────┘            │    │
│  └─────────────────────────────────────────────────────────────────┘    │
│  ┌─────────────────────────────────────────────────────────────────┐    │
│  │                      SERVICE ADAPTERS                           │    │
│  │  Secrets │ Storage │ Artifacts │ Trace │ Terminal │ Logs        │    │
│  └─────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
┌───────────────────────────────────▼─────────────────────────────────────┐
│                           INFRASTRUCTURE                                │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐       │
│  │PostgreSQL│ │  MinIO   │ │  Redis   │ │Redpanda  │ │   Loki   │       │
│  │  (Data)  │ │ (Files)  │ │(Terminal)│ │ (Kafka)  │ │  (Logs)  │       │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘       │
└─────────────────────────────────────────────────────────────────────────┘
```

***

## Technology Stack

| Layer              | Stack                                                                           |
| ------------------ | ------------------------------------------------------------------------------- |
| **Frontend**       | React 19, TypeScript, Vite, TailwindCSS, Radix UI, ReactFlow, xterm.js, Zustand |
| **Backend**        | NestJS, TypeScript, Bun, Drizzle ORM, Clerk Auth                                |
| **Worker**         | Node.js, TypeScript, Temporal SDK, Docker                                       |
| **Infrastructure** | PostgreSQL 16, Temporal, MinIO, Redis, Redpanda (Kafka), Loki                   |

***

## Core Deep-Dives

To keep this guide concise, complicated subsystems are documented in their own dedicated files:

* **[Workflow Compilation (DSL)](/architecture/workflow-compilation)**: How visual graphs are transformed into executable instructions.
* **[Temporal Orchestration](/architecture/temporal-orchestration)**: How we use Temporal for durability and worker scaling.
* **[Streaming Pipelines](/architecture/streaming-pipelines)**: How terminal, logs, and events are delivered in real-time.
* **[Human-in-the-Loop](/architecture/human-in-the-loop)**: How we pause workflows for manual approvals and forms.

***

## Component Categories

Components are the building blocks of workflows:

| Category          | Description                             | Examples                                                |
| ----------------- | --------------------------------------- | ------------------------------------------------------- |
| **security**      | Security scanning and enumeration tools | Subfinder, DNSX, Nuclei, Naabu, HTTPx, TruffleHog       |
| **ai**            | AI/ML and agent components              | LLM Generate, AI Agent, MCP Providers                   |
| **core**          | Utility and data processing             | HTTP Request, File Loader, Logic Script, JSON Transform |
| **notification**  | Alerts and messaging                    | Slack, Email                                            |
| **manual-action** | Human-in-the-loop                       | Approvals, Forms, Selection                             |
| **github**        | GitHub integrations                     | Remove Org Membership                                   |

***

## Security Architecture

### Authentication & Multi-tenancy

* **Clerk Integration** — Production-ready authentication for hosted environments.
* **Local Auth** — Default for local setup using `ADMIN_USERNAME` / `ADMIN_PASSWORD`.
* **Organization Isolation** — All data scoped by `organization_id`.

### Secrets Management

* **AES-256-GCM** encryption at rest.
* **Versioned secrets** with active/inactive tracking.
* **Master key** via `SECRET_STORE_MASTER_KEY` environment variable.

### Container Isolation

* **IsolatedContainerVolume** — Per-tenant, per-run Docker volumes. See **[Isolated Volumes](/development/isolated-volumes)**.
* **Network isolation** — Components run with `network: none` or `bridge`.
* **Automatic cleanup** — Volumes destroyed after execution.

***

## Development URLs

All application services are accessible through nginx on port 80:

| Service          | URL                                                        |
| ---------------- | ---------------------------------------------------------- |
| Frontend         | [http://localhost/](http://localhost/)                     |
| Backend API      | [http://localhost/api/](http://localhost/api/)             |
| Analytics        | [http://localhost/analytics/](http://localhost/analytics/) |
| Temporal UI      | [http://localhost:8081](http://localhost:8081)             |
| MinIO Console    | [http://localhost:9001](http://localhost:9001)             |
| Redpanda Console | [http://localhost:8082](http://localhost:8082)             |
| Loki             | [http://localhost:3100](http://localhost:3100)             |

<Note>
  Individual service ports (5173, 3211, 5601) are available for debugging but should not be used in normal development. All traffic flows through nginx on port 80.
</Note>

***

## Learn More

* **Component Development**: `/development/component-development`
* **Getting Started**: `/getting-started`


# Command Reference
Source: https://docs.shipsec.ai/command-reference

Complete reference for just commands and development utilities

ShipSec Studio uses [just](https://github.com/casey/just) as its command runner. Run `just` or `just help` to see all available commands.

***

## Getting Started

### Initialize Project

```bash theme={null}
just init
```

Sets up the development environment:

* Installs dependencies with Bun
* Creates environment files from examples (`backend/.env`, `worker/.env`, `frontend/.env`)

***

## Development Commands

### Start Development Environment

```bash theme={null}
just dev
```

Starts the full development environment with hot-reload:

* Checks for environment files
* Starts Docker infrastructure (Postgres, Temporal, MinIO, Loki)
* Waits for services to be ready
* Runs database migrations
* Starts backend, worker, and frontend via PM2

**Access points (all via nginx on port 80):**

| Service     | URL                                                        |
| ----------- | ---------------------------------------------------------- |
| Frontend    | [http://localhost/](http://localhost/)                     |
| Backend API | [http://localhost/api/](http://localhost/api/)             |
| Analytics   | [http://localhost/analytics/](http://localhost/analytics/) |
| Temporal UI | [http://localhost:8081](http://localhost:8081)             |

### Stop Development

```bash theme={null}
just dev stop
```

Stops all PM2 processes and Docker infrastructure.

### View Logs

```bash theme={null}
just dev logs
```

Streams logs from all application processes via PM2.

### Check Status

```bash theme={null}
just dev status
```

Shows status of PM2 processes and Docker containers.

***

## Production Commands

### Start Production

```bash theme={null}
just prod
```

Starts the production environment using Docker Compose with pre-built images.

**Access points (all via nginx on port 80):**

| Service     | URL                                                        |
| ----------- | ---------------------------------------------------------- |
| Frontend    | [http://localhost/](http://localhost/)                     |
| Backend API | [http://localhost/api/](http://localhost/api/)             |
| Analytics   | [http://localhost/analytics/](http://localhost/analytics/) |
| Temporal UI | [http://localhost:8081](http://localhost:8081)             |

### Build and Start

```bash theme={null}
just prod build
```

Builds Docker images and starts the production environment.

### Stop Production

```bash theme={null}
just prod stop
```

Stops all production containers.

### View Production Logs

```bash theme={null}
just prod logs
```

Streams logs from all production containers.

### Check Production Status

```bash theme={null}
just prod status
```

Shows status of production containers.

### Clean Production

<Danger>This deletes all data.</Danger>

```bash theme={null}
just prod clean
```

Stops containers, removes volumes, and prunes Docker system.

***

## Infrastructure Commands

### Start Infrastructure

```bash theme={null}
just infra up
```

Starts only the infrastructure containers:

* PostgreSQL
* Temporal
* MinIO
* Redis
* Loki

### Stop Infrastructure

```bash theme={null}
just infra down
```

Stops all infrastructure containers.

### View Infrastructure Logs

```bash theme={null}
just infra logs
```

Streams logs from infrastructure containers.

### Clean Infrastructure

```bash theme={null}
just infra clean
```

Stops infrastructure and removes all volumes.

<Danger>This deletes all data.</Danger>

***

## Utility Commands

### Show All Status

```bash theme={null}
just status
```

Comprehensive status check showing:

* PM2 services
* Infrastructure containers
* Production containers

### Reset Database

```bash theme={null}
just db-reset
```

Drops and recreates the database, then runs migrations. **This deletes all data.**

<Warning>
  This command requires PostgreSQL to be running (`just dev` or `just infra up`).
</Warning>

### Build Images Only

```bash theme={null}
just build
```

Builds production Docker images without starting containers.

***

## Quick Reference

| Command            | Description                |
| ------------------ | -------------------------- |
| `just init`        | Initialize environment     |
| `just dev`         | Start development          |
| `just dev stop`    | Stop development           |
| `just dev logs`    | View dev logs              |
| `just dev status`  | Check dev status           |
| `just prod`        | Start production           |
| `just prod build`  | Build and start production |
| `just prod stop`   | Stop production            |
| `just prod logs`   | View production logs       |
| `just prod clean`  | Clean production data      |
| `just infra up`    | Start infrastructure       |
| `just infra down`  | Stop infrastructure        |
| `just infra logs`  | View infrastructure logs   |
| `just infra clean` | Clean infrastructure data  |
| `just status`      | Show all status            |
| `just db-reset`    | Reset database             |
| `just build`       | Build images               |
| `just help`        | Show help                  |

***

## Environment Files

The `just init` command creates these environment files:

| File            | Purpose                   |
| --------------- | ------------------------- |
| `backend/.env`  | Backend API configuration |
| `worker/.env`   | Worker configuration      |
| `frontend/.env` | Frontend configuration    |

Each file is created from its corresponding `.env.example` template.


# AI Components
Source: https://docs.shipsec.ai/components/ai

LLM integrations for intelligent workflow automation

AI components in ShipSec Studio follow a **Provider-Consumer** architecture.

1. **Providers**: Handle credentials, model selection, and API configuration (OpenAI, Gemini, OpenRouter).
2. **Consumers**: Execute specific tasks (Text Generation or Autonomous Agents) using the configuration emitted by a Provider.

***

## Providers

Provider nodes normalize credentials and model settings into a reusable **LLM Provider Config**.

### OpenAI Provider

Configures access to OpenAI or OpenAI-compatible endpoints.

| Input    | Type   | Description                                     |
| -------- | ------ | ----------------------------------------------- |
| `apiKey` | Secret | OpenAI API key (typically from a Secret Loader) |

| Parameter    | Type   | Description                                 |
| ------------ | ------ | ------------------------------------------- |
| `model`      | Select | `gpt-5.2`, `gpt-5.1`, `gpt-5`, `gpt-5-mini` |
| `apiBaseUrl` | Text   | Optional override for the API base URL      |

### Gemini Provider

Configures access to Google's Gemini models.

| Input    | Type   | Description       |
| -------- | ------ | ----------------- |
| `apiKey` | Secret | Google AI API key |

| Parameter    | Type   | Description                                                        |
| ------------ | ------ | ------------------------------------------------------------------ |
| `model`      | Select | `gemini-3-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-pro` |
| `apiBaseUrl` | Text   | Optional override for the API base URL                             |
| `projectId`  | Text   | Optional Google Cloud project identifier                           |

### OpenRouter Provider

Configures access to multiple LLM providers through OpenRouter's unified API.

| Input    | Type   | Description        |
| -------- | ------ | ------------------ |
| `apiKey` | Secret | OpenRouter API key |

| Parameter     | Type | Description                                                         |
| ------------- | ---- | ------------------------------------------------------------------- |
| `model`       | Text | Model slug (e.g., `openrouter/auto`, `anthropic/claude-3.5-sonnet`) |
| `apiBaseUrl`  | Text | Optional override for the API base URL                              |
| `httpReferer` | Text | Application URL for OpenRouter ranking                              |
| `appTitle`    | Text | Application title for OpenRouter ranking                            |

***

## Consumers

Consumer nodes perform the actual AI work. They require a **Provider Config** output from one of the providers above.

### AI Generate Text

Performs a one-shot chat completion.

| Input         | Type       | Description                                             |
| ------------- | ---------- | ------------------------------------------------------- |
| `userPrompt`  | Text       | The primary request or data to process                  |
| `chatModel`   | Credential | **Required.** Connect a Provider output here            |
| `modelApiKey` | Secret     | Optional. supersedes the API key in the Provider Config |

| Parameter      | Type     | Description                                  |
| -------------- | -------- | -------------------------------------------- |
| `systemPrompt` | Textarea | Instructions that guide the model's behavior |
| `temperature`  | Number   | Creativity vs. determinism (0.0 to 2.0)      |
| `maxTokens`    | Number   | Maximum tokens to generate                   |

| Output         | Type | Description                     |
| -------------- | ---- | ------------------------------- |
| `responseText` | Text | The assistant's response        |
| `usage`        | JSON | Token consumption metadata      |
| `rawResponse`  | JSON | Full API response for debugging |

***

### AI SDK Agent

An autonomous agent that uses reasoning steps and tool-calling to solve complex tasks.

| Input               | Type       | Description                                       |
| ------------------- | ---------- | ------------------------------------------------- |
| `userInput`         | Text       | The task or question for the agent                |
| `chatModel`         | Credential | **Required.** Connect a Provider output here      |
| `conversationState` | JSON       | Optional. Connect from a previous turn for memory |
| `mcpTools`          | List       | Optional. Connect tools from MCP Providers        |

| Parameter                 | Type     | Description                                                               |
| ------------------------- | -------- | ------------------------------------------------------------------------- |
| `systemPrompt`            | Textarea | Core identity and constraints for the agent                               |
| `temperature`             | Number   | Reasoning creativity (default 0.7)                                        |
| `stepLimit`               | Number   | Max "Think -> Act -> Observe" loops (1-12)                                |
| `memorySize`              | Number   | Number of previous turns to retain in context                             |
| `structuredOutputEnabled` | Toggle   | Enable to enforce a specific JSON output structure                        |
| `schemaType`              | Select   | How to define the schema: `json-example` or `json-schema`                 |
| `jsonExample`             | JSON     | Example JSON object for schema inference (all properties become required) |
| `jsonSchema`              | JSON     | Full JSON Schema definition for precise validation                        |
| `autoFixFormat`           | Toggle   | Attempt to extract valid JSON from malformed responses                    |

| Output              | Type | Description                                        |
| ------------------- | ---- | -------------------------------------------------- |
| `responseText`      | Text | Final answer after reasoning is complete           |
| `structuredOutput`  | JSON | Parsed structured output (when enabled)            |
| `conversationState` | JSON | Updated state to pass to the next agent node       |
| `reasoningTrace`    | JSON | Detailed step-by-step logs of the agent's thoughts |
| `agentRunId`        | Text | Unique session ID for tracking and streaming       |

***

## MCP Tools (Model Context Protocol)

ShipSec Studio supports the **Model Context Protocol (MCP)**, allowing AI agents to interact with external tools over HTTP.

### MCP HTTP Tools

Exposes a set of tools from a remote HTTP server that implements the MCP contract.

| Input         | Type | Description                                         |
| ------------- | ---- | --------------------------------------------------- |
| `endpoint`    | Text | The HTTP URL where the MCP server is hosted         |
| `headersJson` | Text | Optional JSON of headers (e.g., Auth tokens)        |
| `tools`       | JSON | List of tool definitions available on that endpoint |

| Parameter  | Type | Description                                        |
| ---------- | ---- | -------------------------------------------------- |
| `endpoint` | Text | Destination URL for tool execution                 |
| `tools`    | JSON | Array of tools with `id`, `title`, and `arguments` |

| Output  | Type | Description                                      |
| ------- | ---- | ------------------------------------------------ |
| `tools` | List | Normalized MCP tool definitions for the AI Agent |

### MCP Tool Merge

Combines multiple MCP tool lists into a single consolidated list.

| Input              | Type | Description                        |
| ------------------ | ---- | ---------------------------------- |
| `toolsA`, `toolsB` | List | Multiple upstream MCP tool outputs |

| Parameter | Type | Description                                  |
| --------- | ---- | -------------------------------------------- |
| `slots`   | JSON | Configure additional input ports for merging |

| Output  | Type | Description                                    |
| ------- | ---- | ---------------------------------------------- |
| `tools` | List | De-duplicated list of tools ready for an agent |

***

## Use Cases

### Automated Alert Triage

**Flow:** `Provider` → `AI Generate Text`

Analyze incoming security alerts to filter out false positives.
**Prompt:** "Given this alert payload: , determine if it's a real threat or noise."

### Investigative Agent

**Flow:** `Provider` + `MCP Tool` → `AI Agent`

An agent that searches through logs and performs lookups to investigate a specific IP address.
**Task:** "Investigate the IP  using the available Splunk and VirusTotal tools."

### Structured Output for Data Extraction

**Flow:** `Provider` → `AI Agent` (with Structured Output enabled)

Extract structured data from unstructured security reports. Enable **Structured Output** and provide a JSON example:

```json theme={null}
{
  "severity": "high",
  "affected_systems": ["web-server-01"],
  "remediation_steps": ["Patch CVE-2024-1234", "Restart service"]
}
```

The agent will always return validated JSON matching this schema, ready for downstream processing.

***

## Best Practices

<Note>
  **The Provider Concept**: Always place a Provider node (OpenAI/Gemini/OpenRouter) at the start of your AI chain. This allows you to swap models or providers for the entire workflow by changing just one node.
</Note>

### Prompt Engineering

1. **Use Structured Output**: When you need consistent JSON for downstream nodes, enable **Structured Output** instead of relying on prompt instructions. This guarantees schema compliance and eliminates parsing errors.
2. **Use System Prompts**: Set high-level rules (e.g., "You are a senior security researcher") in the System Prompt parameter instead of the User Input.
3. **Variable Injection**: Use `{{variableName}}` syntax to inject data from upstream nodes into your prompts.

### Memory & State

For multi-turn conversations, always loop the `conversationState` output of the AI Agent back into the `conversationState` input of the next agent invocation (or store it in a persistent variable).


# Core Components
Source: https://docs.shipsec.ai/components/core

Essential building blocks for workflow inputs, outputs, and data transformation

Core components handle workflow triggers, file operations, data transformation, and output destinations.

***

## Triggers

### Manual Trigger

Starts a workflow manually. Configure runtime inputs to collect data (files, text, etc.) when triggered.

| Parameter       | Type | Description                         |
| --------------- | ---- | ----------------------------------- |
| `runtimeInputs` | JSON | Define inputs to collect at runtime |

**Supported input types:** `file`, `text`, `number`, `json`, `array`

**Example use cases:**

* Collect uploaded scope files before running security scans
* Prompt operators for target domains or API keys

***

### Webhook

Sends JSON payloads to external HTTP endpoints with retries and timeouts.

| Parameter   | Type   | Description                      |
| ----------- | ------ | -------------------------------- |
| `url`       | URL    | Destination endpoint             |
| `method`    | Select | POST, PUT, or PATCH              |
| `payload`   | JSON   | Request body                     |
| `headers`   | JSON   | HTTP headers                     |
| `timeoutMs` | Number | Request timeout (default: 30000) |
| `retries`   | Number | Retry attempts (default: 3)      |

**Example use cases:**

* Send scan results to Slack or Teams
* POST assets to a custom API

***

## File Operations

### File Loader

Loads file content from storage for use in workflows.

| Input    | Type | Description                |
| -------- | ---- | -------------------------- |
| `fileId` | UUID | File ID from uploaded file |

| Output        | Type   | Description                    |
| ------------- | ------ | ------------------------------ |
| `file`        | Object | File metadata + base64 content |
| `textContent` | String | Decoded UTF-8 text             |

***

### Text Splitter

Splits text into an array of strings by separator.

| Parameter   | Type        | Description                     |
| ----------- | ----------- | ------------------------------- |
| `text`      | String/File | Text content to split           |
| `separator` | String      | Split character (default: `\n`) |

| Output  | Type   | Description     |
| ------- | ------ | --------------- |
| `items` | Array  | Split strings   |
| `count` | Number | Number of items |

**Example:** Split newline-delimited subdomains before passing to scanners.

***

### Text Joiner

Joins array elements into a single string.

| Parameter   | Type   | Description                    |
| ----------- | ------ | ------------------------------ |
| `items`     | Array  | Array of strings to join       |
| `separator` | String | Join character (default: `\n`) |

| Output | Type   | Description   |
| ------ | ------ | ------------- |
| `text` | String | Joined string |

***

## Secrets

### Secret Loader

Fetches secrets from the ShipSec-managed secret store.

| Parameter      | Type   | Description          |
| -------------- | ------ | -------------------- |
| `secretName`   | Secret | Secret name or UUID  |
| `version`      | Number | Optional version pin |
| `outputFormat` | Select | `raw` or `json`      |

| Output     | Type   | Description                            |
| ---------- | ------ | -------------------------------------- |
| `secret`   | Any    | Resolved secret value (masked in logs) |
| `metadata` | Object | Secret version info                    |

<Note>
  Secret values are automatically masked in all logs and terminal output.
</Note>

***

## Data Transformation

### Array Pick

Extracts specific items from an array by index.

| Parameter | Type  | Description     |
| --------- | ----- | --------------- |
| `array`   | Array | Source array    |
| `indices` | Array | Indices to pick |

| Output   | Type  | Description    |
| -------- | ----- | -------------- |
| `picked` | Array | Selected items |

### Array Pack

Combines multiple values into a single array.

| Parameter | Type   | Description    |
| --------- | ------ | -------------- |
| `values`  | Any\[] | Values to pack |

| Output  | Type  | Description  |
| ------- | ----- | ------------ |
| `array` | Array | Packed array |

### Console Log

Outputs data to workflow logs for debugging.

| Parameter | Type   | Description    |
| --------- | ------ | -------------- |
| `data`    | Any    | Data to log    |
| `label`   | String | Optional label |

***

## Storage Destinations

### Artifact Writer

Writes workflow artifacts to ShipSec storage.

| Parameter  | Type   | Description       |
| ---------- | ------ | ----------------- |
| `content`  | Any    | Content to store  |
| `filename` | String | Artifact filename |
| `mimeType` | String | Content type      |

| Output       | Type   | Description        |
| ------------ | ------ | ------------------ |
| `artifactId` | UUID   | Stored artifact ID |
| `url`        | String | Download URL       |

### File Writer

Writes content to a file in workflow storage.

| Parameter | Type   | Description  |
| --------- | ------ | ------------ |
| `content` | String | File content |
| `path`    | String | File path    |

### Destination S3

Uploads files to an S3-compatible bucket.

| Parameter     | Type   | Description     |
| ------------- | ------ | --------------- |
| `bucket`      | String | S3 bucket name  |
| `key`         | String | Object key      |
| `content`     | Buffer | File content    |
| `credentials` | Object | AWS credentials |

### AWS Credentials

Provides AWS credentials for S3 operations.

| Parameter         | Type   | Description           |
| ----------------- | ------ | --------------------- |
| `accessKeyId`     | Secret | AWS Access Key ID     |
| `secretAccessKey` | Secret | AWS Secret Access Key |
| `region`          | String | AWS region            |

| Output        | Type   | Description                         |
| ------------- | ------ | ----------------------------------- |
| `credentials` | Object | Credential object for S3 components |

***

## Analytics

### Analytics Sink

Indexes workflow output data into OpenSearch for analytics dashboards, queries, and alerts. Connect the `results` port from upstream security scanners.

| Input  | Type | Description                                                               |
| ------ | ---- | ------------------------------------------------------------------------- |
| `data` | Any  | Data to index. Works best with `list<json>` from scanner `results` ports. |

| Output          | Type    | Description                           |
| --------------- | ------- | ------------------------------------- |
| `indexed`       | Boolean | Whether data was successfully indexed |
| `documentCount` | Number  | Number of documents indexed           |
| `indexName`     | String  | Name of the OpenSearch index used     |

| Parameter             | Type    | Description                                                                                                          |
| --------------------- | ------- | -------------------------------------------------------------------------------------------------------------------- |
| `indexSuffix`         | String  | Custom suffix for the index name. Defaults to slugified workflow name.                                               |
| `assetKeyField`       | Select  | Field to use as asset identifier. Options: auto, asset\_key, host, domain, subdomain, url, ip, asset, target, custom |
| `customAssetKeyField` | String  | Custom field name when assetKeyField is "custom"                                                                     |
| `failOnError`         | Boolean | When enabled, workflow stops if indexing fails. Default: false (fire-and-forget)                                     |

**How it works:**

1. Each item in the input array becomes a separate document
2. Workflow context is added under `shipsec.*` namespace
3. Nested objects are serialized to JSON strings (prevents field explosion)
4. All documents get the same `@timestamp`

**Example use cases:**

* Index Nuclei scan results for trend analysis
* Store TruffleHog secrets for tracking over time
* Aggregate vulnerability data across workflows

<Note>
  See [Workflow Analytics](/development/workflow-analytics) for detailed setup and querying guide.
</Note>


# Components Overview
Source: https://docs.shipsec.ai/components/overview

Drag-and-drop building blocks for security automation workflows

Components are the building blocks of ShipSec Studio workflows. Each component performs a specific task and can be connected together to create powerful automation pipelines.

## Component Categories

<CardGroup>
  <Card title="Core" icon="cube" href="/components/core">
    Triggers, file handling, data transformation, and outputs
  </Card>

  <Card title="Security" icon="shield" href="/components/security">
    Subdomain discovery, port scanning, DNS resolution, secret detection
  </Card>

  <Card title="AI" icon="brain" href="/components/ai">
    Provider configurations and autonomous agents
  </Card>
</CardGroup>

## How Components Work

Each component has:

* **Inputs** – Data ports that accept connections from other components
* **Outputs** – Data ports that can be connected to downstream components
* **Parameters** – Configurable settings in the sidebar

Components run inside Docker containers for isolation and reliability. Workflows are orchestrated by [Temporal](https://temporal.io/) with automatic retries and resumability.

## Component Structure

```typescript theme={null}
interface ComponentDefinition<Input, Output> {
  id: string;           // Unique identifier
  label: string;        // Display name
  category: string;     // Category for grouping
  runner: RunnerConfig; // Execution configuration
  inputSchema: ZodSchema<Input>;
  outputSchema: ZodSchema<Output>;
  execute: (input: Input, context: ExecutionContext) => Promise<Output>;
}
```

## Example Pipelines

### Attack Surface Discovery

```
Manual Trigger → Subfinder → DNSx → httpx → Notify
```

1. **Manual Trigger** – User provides target domains
2. **Subfinder** – Discovers subdomains
3. **DNSx** – Resolves DNS records
4. **httpx** – Probes for live HTTP services
5. **Notify** – Sends results to Slack, Discord, or Teams

### Secret Detection

```
Manual Trigger → TruffleHog → OpenAI Provider → AI Generate Text → Notify
```

1. **Manual Trigger** – User provides repository URL
2. **TruffleHog** – Scans for leaked credentials
3. **OpenAI Provider** – Set up model and API key
4. **AI Generate Text** – Analyzes and prioritizes findings
5. **Notify** – Alerts team via Slack, Discord, or email

## Execution Context

Components receive an `ExecutionContext` with access to:

| Service     | Description                   |
| ----------- | ----------------------------- |
| `storage`   | File upload/download to MinIO |
| `secrets`   | Encrypted secrets access      |
| `artifacts` | Workflow artifact storage     |
| `trace`     | Event recording               |
| `logger`    | Structured logging            |
| `terminal`  | Terminal output streaming     |

## Building Custom Components

See the [Component Development Guide](/development/component-development) for detailed instructions on building your own components.


# Security Components
Source: https://docs.shipsec.ai/components/security

Open source security tools for security automation workflows

Security components wrap popular open-source tools for subdomain discovery, DNS resolution, port scanning, and secret detection.

***

## Subdomain Discovery

### Subfinder

<Info>
  [GitHub](https://github.com/projectdiscovery/subfinder) · Docker: `ghcr.io/shipsecai/subfinder`
</Info>

Discovers subdomains using passive sources.

| Input            | Type   | Description                                             |
| ---------------- | ------ | ------------------------------------------------------- |
| `domains`        | Array  | Target domains to enumerate                             |
| `providerConfig` | Secret | Optional provider-config.yaml for authenticated sources |

| Output           | Type   | Description                    |
| ---------------- | ------ | ------------------------------ |
| `subdomains`     | Array  | Discovered subdomain hostnames |
| `rawOutput`      | String | Raw tool output                |
| `subdomainCount` | Number | Total subdomains found         |

**Example command:** `subfinder -d example.com -silent`

***

### Amass

<Info>
  [GitHub](https://github.com/owasp-amass/amass) · Docker: `ghcr.io/shipsecai/amass`
</Info>

Active and passive subdomain enumeration.

| Input     | Type   | Description           |
| --------- | ------ | --------------------- |
| `domains` | Array  | Target domains        |
| `mode`    | Select | `passive` or `active` |

| Parameter | Type   | Description                |
| --------- | ------ | -------------------------- |
| `timeout` | Number | Timeout in minutes         |
| `maxDns`  | Number | Max DNS queries per second |

***

### ShuffleDNS + MassDNS

<Info>
  [GitHub (ShuffleDNS)](https://github.com/projectdiscovery/shuffledns) · [GitHub (MassDNS)](https://github.com/blechschmidt/massdns) · Docker: `ghcr.io/shipsecai/shuffledns-massdns`
</Info>

High-performance DNS bruteforcing and resolution. This is a combined image that has both ShuffleDNS with MassDNS pre-installed.

| Input       | Type  | Description               |
| ----------- | ----- | ------------------------- |
| `domains`   | Array | Target domains            |
| `wordlist`  | File  | Wordlist for bruteforcing |
| `resolvers` | Array | Custom DNS resolvers      |

***

## DNS Resolution

### DNSX

<Info>
  [GitHub](https://github.com/projectdiscovery/dnsx) · Docker: `ghcr.io/shipsecai/dnsx`
</Info>

Resolves DNS records with support for multiple record types and custom resolvers.

| Input         | Type  | Description                                  |
| ------------- | ----- | -------------------------------------------- |
| `domains`     | Array | Domains to resolve                           |
| `recordTypes` | Array | DNS types: A, AAAA, CNAME, MX, NS, TXT, etc. |
| `resolvers`   | Array | Custom resolver IPs (e.g., 1.1.1.1:53)       |

| Parameter    | Type    | Description                       |
| ------------ | ------- | --------------------------------- |
| `threads`    | Number  | Concurrent workers (default: 100) |
| `retryCount` | Number  | Retry attempts (default: 2)       |
| `rateLimit`  | Number  | Requests per second               |
| `showCdn`    | Boolean | Annotate CDN providers            |
| `showAsn`    | Boolean | Include ASN info                  |

| Output          | Type   | Description                          |
| --------------- | ------ | ------------------------------------ |
| `results`       | Array  | DNS responses grouped by record type |
| `resolvedHosts` | Array  | Unique resolved hostnames            |
| `rawOutput`     | String | Raw JSONL output                     |

***

## HTTP Probing

### httpx

<Info>
  [GitHub](https://github.com/projectdiscovery/httpx) · Docker: `ghcr.io/shipsecai/httpx`
</Info>

Probes hosts for live HTTP services and captures response metadata.

| Input     | Type  | Description                |
| --------- | ----- | -------------------------- |
| `targets` | Array | Hostnames or URLs to probe |

| Parameter         | Type    | Description                                 |
| ----------------- | ------- | ------------------------------------------- |
| `ports`           | String  | Comma-separated ports (e.g., "80,443,8080") |
| `statusCodes`     | String  | Filter by HTTP status codes                 |
| `threads`         | Number  | Concurrency level                           |
| `followRedirects` | Boolean | Follow HTTP redirects                       |
| `tlsProbe`        | Boolean | Probe TLS endpoints                         |
| `path`            | String  | Specific path to probe                      |

| Output      | Type   | Description            |
| ----------- | ------ | ---------------------- |
| `results`   | Array  | HTTP response metadata |
| `rawOutput` | String | Raw httpx JSON lines   |

**Example command:** `httpx -l targets.txt -json -status-code 200,301`

***

## Port Scanning

### Naabu

<Info>
  [GitHub](https://github.com/projectdiscovery/naabu) · Docker: `ghcr.io/shipsecai/naabu`
</Info>

Fast active port scanning using SYN/CONNECT probes.

| Input     | Type  | Description              |
| --------- | ----- | ------------------------ |
| `targets` | Array | Hostnames or IPs to scan |

| Parameter    | Type    | Description                             |
| ------------ | ------- | --------------------------------------- |
| `ports`      | String  | Custom ports (e.g., "80,443,1000-2000") |
| `topPorts`   | Number  | Scan top N common ports                 |
| `rate`       | Number  | Packets per second                      |
| `retries`    | Number  | Retry attempts per port                 |
| `enablePing` | Boolean | Ping probe before scanning              |

| Output          | Type   | Description            |
| --------------- | ------ | ---------------------- |
| `findings`      | Array  | Open ports per target  |
| `openPortCount` | Number | Total open ports found |

**Example command:** `naabu -host scanme.sh -top-ports 100`

***

## Vulnerability Scanning

### Nuclei

<Info>
  [GitHub](https://github.com/shipsecai/tools/pkgs/container/nuclei) · Docker: `ghcr.io/shipsecai/nuclei`
</Info>

Template-based vulnerability scanning. This is nuclei custom image with nuclei-templates baked in.

| Input       | Type  | Description           |
| ----------- | ----- | --------------------- |
| `targets`   | Array | URLs or hosts to scan |
| `templates` | Array | Template IDs or paths |

| Parameter     | Type   | Description                                      |
| ------------- | ------ | ------------------------------------------------ |
| `severity`    | Array  | Filter by severity (critical, high, medium, low) |
| `rate`        | Number | Requests per second                              |
| `concurrency` | Number | Parallel template executions                     |

| Output          | Type   | Description              |
| --------------- | ------ | ------------------------ |
| `findings`      | Array  | Detected vulnerabilities |
| `criticalCount` | Number | Critical findings count  |

***

## Secret Detection

### TruffleHog

<Info>
  [GitHub](https://github.com/trufflesecurity/trufflehog) · Docker: `ghcr.io/shipsecai/trufflehog`
</Info>

Scans for leaked credentials across repositories, filesystems, and cloud storage.

| Input        | Type   | Description                                 |
| ------------ | ------ | ------------------------------------------- |
| `scanTarget` | String | Repository URL, path, bucket, or image      |
| `scanType`   | Select | git, github, gitlab, s3, filesystem, docker |

| Parameter      | Type    | Description                                    |
| -------------- | ------- | ---------------------------------------------- |
| `onlyVerified` | Boolean | Show only verified secrets (default: true)     |
| `branch`       | String  | Specific branch to scan                        |
| `sinceCommit`  | String  | Scan commits since reference (for PR scanning) |

| Output               | Type    | Description                               |
| -------------------- | ------- | ----------------------------------------- |
| `secrets`            | Array   | Detected secrets with verification status |
| `verifiedCount`      | Number  | Number of verified secrets                |
| `hasVerifiedSecrets` | Boolean | Alert flag                                |

**Example command:** `trufflehog git https://github.com/org/repo --results=verified --json`

***

## Cloud Security

### Prowler Scan

<Info>
  [GitHub](https://github.com/prowler-cloud/prowler) · Docker: `ghcr.io/shipsecai/prowler`
</Info>

Cloud (AWS, Azure, GCP) security posture management. Best practices auditing.

| Input         | Type   | Description            |
| ------------- | ------ | ---------------------- |
| `credentials` | Object | AWS credentials        |
| `checks`      | Array  | Specific checks to run |

| Parameter  | Type  | Description           |
| ---------- | ----- | --------------------- |
| `severity` | Array | Filter by severity    |
| `services` | Array | AWS services to audit |

### Supabase Scanner

<Info>
  [GitHub](https://ghcr.io/shipsecai/supabase-scanner) · Docker: `ghcr.io/shipsecai/supabase-scanner`
</Info>

Scans Supabase instances for misconfigurations.

| Input        | Type   | Description          |
| ------------ | ------ | -------------------- |
| `projectUrl` | String | Supabase project URL |
| `apiKey`     | Secret | Supabase API key     |

***

## Notifications

### Notify

<Info>
  [GitHub](https://github.com/projectdiscovery/notify) · Docker: `ghcr.io/shipsecai/notify`
</Info>

Sends alerts to Slack, Discord, Telegram, or email.

| Input      | Type   | Description                     |
| ---------- | ------ | ------------------------------- |
| `message`  | String | Alert message                   |
| `provider` | Select | slack, discord, telegram, email |
| `config`   | Secret | Provider configuration          |

***

## Example Workflow

A complete attack surface discovery pipeline:

```
Manual Trigger (domains input)
    ↓
Subfinder (subdomain discovery)
    ↓
DNSx (DNS resolution)
    ↓
httpx (HTTP probing)
    ↓
Nuclei (vulnerability scanning)
    ↓
Notify (notify team)
```


# Product Analytics (PostHog)
Source: https://docs.shipsec.ai/development/analytics

PostHog integration for product analytics and session recording

ShipSec Studio uses PostHog for product analytics and session recording in the frontend. The integration is gated so local clones without environment variables continue to work without errors.

***

## Environment Variables

Frontend variables (set in your hosting provider):

| Variable                   | Description                                     |
| -------------------------- | ----------------------------------------------- |
| `VITE_PUBLIC_POSTHOG_KEY`  | Project API key                                 |
| `VITE_PUBLIC_POSTHOG_HOST` | PostHog host (e.g., `https://us.i.posthog.com`) |

See `frontend/.env.example` for a template. If these are not set, analytics is fully disabled at runtime (no client init, helpers no-op).

***

## Initialization

`frontend/src/main.tsx` initializes the global `posthog` client via `posthog.init(...)` and mounts `PostHogProvider` when both variables are present.

Configuration:

* Session recording enabled with privacy defaults
* `maskAllInputs: true` - All input fields masked
* `maskAllText: false` - On-screen text visible for useful context
* Exceptions captured automatically
* Pageviews captured by router listener

***

## SPA Pageviews

`frontend/src/features/analytics/AnalyticsRouterListener.tsx` captures `$pageview` on `react-router` navigation. It checks `isAnalyticsEnabled()` before sending.

***

## User Identification

`frontend/src/features/analytics/PostHogClerkBridge.tsx` bridges Clerk auth to PostHog (only when analytics is enabled and Clerk is the active provider):

* Calls `posthog.identify(user.id, { email, name, username })`
* Sets the `organization` group when available
* Calls `posthog.reset()` on sign-out

***

## Event Taxonomy

| Event                        | Description                      | Properties                                |
| ---------------------------- | -------------------------------- | ----------------------------------------- |
| `ui_workflow_list_viewed`    | Workflow list loads              | `workflows_count?`                        |
| `ui_workflow_create_clicked` | User clicked create workflow CTA | -                                         |
| `ui_workflow_builder_loaded` | Builder opened                   | `workflow_id?`, `is_new`, `node_count?`   |
| `ui_workflow_created`        | After successful create          | `workflow_id`, `node_count`, `edge_count` |
| `ui_workflow_saved`          | After successful update          | `workflow_id`, `node_count`, `edge_count` |
| `ui_workflow_run_started`    | Run kicked off                   | `workflow_id`, `run_id?`, `node_count?`   |
| `ui_node_added`              | Component dropped on canvas      | `workflow_id?`, `component_slug`          |
| `ui_secret_created`          | Secret created                   | `has_tags?`, `tag_count?`, `name_length?` |
| `ui_secret_deleted`          | Secret deleted                   | `name_length?`                            |

Helpers live in `frontend/src/features/analytics/events.ts` and validate payloads with Zod. All helper calls no-op when analytics is disabled.

***

## Privacy & Controls

| Feature               | Implementation                                        |
| --------------------- | ----------------------------------------------------- |
| **Do Not Track**      | Respected via `respect_dnt: true`                     |
| **Session Recording** | Inputs masked, on-screen text unmasked                |
| **Secrets Manager**   | Events never send raw secret identifiers              |
| **Local/Dev Safety**  | Analytics only initializes when both env vars present |

<Note>
  Optional runtime kill-switch can be added later (e.g., `VITE_ENABLE_ANALYTICS=false`).
</Note>

***

## Local Verification

<Steps>
  <Step title="Start the frontend">
    Run the frontend with PostHog environment variables set.
  </Step>

  <Step title="Log in and navigate">
    Log in to the application and navigate between pages.
  </Step>

  <Step title="Verify in PostHog">
    Check PostHog Live Events for `$pageview` events.
  </Step>

  <Step title="Check session recording">
    Confirm a session recording is created and inputs are masked.
  </Step>
</Steps>

***

## Troubleshooting

### Events not arriving

Ensure both env vars are set and `main.tsx` initializes `posthog` (search for `posthog.init`).

### Helpers send but nothing recorded

Confirm provider uses `<PostHogProvider client={posthog}>` (not apiKey prop) so the global singleton is the same instance.

### Compile error '@/config/env'

Ensure `frontend/src/config/env.ts` exists; it provides typed access to optional branch labels used by Sidebar.

***

## Adding New Events

To add a new analytics event:

1. **Define the event schema** in `frontend/src/features/analytics/events.ts`:

```typescript theme={null}
const workflowDuplicatedSchema = z.object({
  workflow_id: z.string(),
  source_workflow_id: z.string(),
});

export function trackWorkflowDuplicated(props: z.infer<typeof workflowDuplicatedSchema>) {
  if (!isAnalyticsEnabled()) return;
  
  const validated = workflowDuplicatedSchema.parse(props);
  posthog.capture('ui_workflow_duplicated', validated);
}
```

2. **Call the helper** in your component:

```typescript theme={null}
import { trackWorkflowDuplicated } from '@/features/analytics/events';

// In your component
trackWorkflowDuplicated({
  workflow_id: newWorkflow.id,
  source_workflow_id: sourceWorkflow.id,
});
```

3. **Document the event** in this page's Event Taxonomy table.

***

## Best Practices

### Do

* ✅ Use typed event helpers with Zod validation
* ✅ Include relevant context (IDs, counts) for analysis
* ✅ Keep property names consistent (`snake_case`)
* ✅ Test events locally before deploying

### Don't

* ❌ Send PII (emails, names) in event properties
* ❌ Send raw secret values or API keys
* ❌ Send sensitive file contents
* ❌ Bypass the `isAnalyticsEnabled()` check


# Component Development
Source: https://docs.shipsec.ai/development/component-development

Build custom security components for ShipSec Studio

This guide covers everything you need to build components for ShipSec Studio.

***

## Getting Started

### File Location

Components live in `worker/src/components/<category>/`:

```
worker/src/components/
├── security/        # Security tools (dnsx, subfinder, nuclei)
├── core/            # Core utilities (http-request, file-loader)
├── ai/              # AI components (llm, agents)
├── notification/    # Notifications (slack, email)
└── manual-action/   # Human-in-the-loop (approvals, forms)
```

### Category Source of Truth

Component categories are defined once in `packages/shared/src/component-categories.ts`.

* Backend categorization and API metadata read from this shared registry.
* Frontend schema validation and category styling also read from the same registry.

When adding or renaming a category, update this shared file so backend and frontend stay in sync.

### ID Naming Convention

```
<namespace>.<tool>.<action>

Examples:
  shipsec.dnsx.run       # Security tool
  core.http.request      # Core utility  
  ai.llm.generate        # AI component
  notification.slack     # Notification
```

***

## Runner Types

| Type     | Use Case                                        | Example                               |
| -------- | ----------------------------------------------- | ------------------------------------- |
| `inline` | Pure TypeScript (HTTP calls, transforms, logic) | FileLoader, WebhookPost, HTTP Request |
| `docker` | CLI tools in containers                         | Subfinder, DNSX, Nuclei               |
| `remote` | External executors (future)                     | K8s jobs, ECS tasks                   |

### Inline Component Example

```typescript theme={null}
import { z } from 'zod';
import { 
  componentRegistry, 
  defineComponent,
  inputs,
  outputs,
  port,
} from '@shipsec/component-sdk';

const definition = defineComponent({
  id: 'core.http.request',
  label: 'HTTP Request',
  category: 'transform',
  runner: { kind: 'inline' },
  
  inputs: inputs({
    url: port(z.string().url(), {
      label: 'URL',
      description: 'The URL to fetch',
    }),
    method: port(z.enum(['GET', 'POST']).default('GET'), {
      label: 'Method',
      description: 'HTTP verb to use',
    }),
  }),
  
  outputs: outputs({
    status: port(z.number(), { label: 'Status Code' }),
    body: port(z.string(), { label: 'Response Body' }),
  }),

  async execute({ inputs }, context) {
    context.logger.info(`Fetching ${inputs.url}`);
    const response = await fetch(inputs.url, { method: inputs.method });
    return { status: response.status, body: await response.text() };
  }
});

export default definition;
```

### Docker Component Example

```typescript theme={null}
import { z } from 'zod';
import { 
  defineComponent,
  inputs,
  outputs,
  port,
  runComponentWithRunner,
  DockerRunnerConfig
} from '@shipsec/component-sdk';

const definition = defineComponent({
  id: 'shipsec.tool.scan',
  label: 'Tool Scanner',
  category: 'security',
  runner: {
    kind: 'docker',
    image: 'tool:latest',
    entrypoint: 'sh',
    command: ['-c', 'tool "$@"', '--'],
    network: 'bridge'
  },
  
  inputs: inputs({
    target: port(z.string(), { label: 'Target Host' }),
  }),
  
  outputs: outputs({
    results: port(z.any(), { label: 'Scan Results' }),
  }),

  async execute({ inputs }, context) {
    const args = ['-json', '-target', inputs.target];
    
    const runnerConfig: DockerRunnerConfig = {
      ...this.runner,
      command: [...(this.runner.command ?? []), ...args],
    };

    const rawOutput = await runComponentWithRunner(
      runnerConfig,
      async (stdout) => {
         // Logic to parse tool stdout
         return JSON.parse(stdout);
      },
      inputs,
      context
    );
    
    return { results: rawOutput };
  }
});

export default definition;
```

***

## ExecutionContext

The `context` passed to `execute()` provides services and utilities:

```typescript theme={null}
async execute({ inputs, params }, context) {
  // Logging (shows in UI timeline)
  context.logger.info('Starting scan...');
  context.logger.warn('Rate limit approaching');
  context.logger.error('Failed to connect');
  
  // Progress events (shows in UI)
  context.emitProgress('Processing 50 targets...');
  
  // Secrets (encrypted, from secret manager)
  const apiKey = await context.secrets?.get('API_KEY');
  
  // File downloads (from MinIO)
  const file = await context.storage?.downloadFile(inputs.fileId);
  
  // Artifact uploads (saved to MinIO, shown in UI)
  await context.artifacts?.upload({
    name: 'report.json',
    content: Buffer.from(JSON.stringify(results)),
    mimeType: 'application/json',
  });
  
  // Run metadata
  const { runId, componentRef } = context;
}
```

***

## Component Definition

A component is defined using `defineComponent` and must specify its `inputs`, `outputs`, and optional `parameters`.

### Inputs vs. Parameters

Understanding the difference between **Inputs** and **Parameters** is critical for building good components.

| Aspect         | Inputs (`inputs()`)                       | Parameters (`parameters()`)               |
| -------------- | ----------------------------------------- | ----------------------------------------- |
| **When set**   | Runtime (during execution)                | Design-time (during workflow building)    |
| **Source**     | Upstream node outputs or Manual Overrides | Sidebar form fields in the UI             |
| **Visibility** | Connection handles on the node            | Config panel in the sidebar               |
| **Use Case**   | Dynamic data (e.g., target IP, file ID)   | Static config (e.g., model name, timeout) |

### Defining Inputs (Ports)

Inputs represent the data that flows into your component from other parts of the workflow. They appear as connection handles on the left side of the node.

```typescript theme={null}
inputs: inputs({
  ipAddress: port(z.string().ip(), { 
    label: 'IP Address',
    description: 'The IP address to scan',
    valuePriority: 'connection-first', // default: connection-first
  }),
})
```

Supported `valuePriority` values:

* `connection-first`: Use the value from the port connection if it exists, otherwise use the manual override.
* `manual-first`: Always use the manual override if a value is provided, even if a port is connected.

### Defining Parameters

Parameters are configuration settings for the component that are set when the user is designing the workflow. They do not accept connections from other nodes; they are always static values (or manual strings).

```typescript theme={null}
parameters: parameters({
  threads: param(z.number().default(10), {
    label: 'Threads',
    editor: 'number',
    min: 1,
    max: 100,
    description: 'Number of concurrent threads to use.'
  }),
  mode: param(z.enum(['fast', 'thorough']).default('fast'), {
    label: 'Scan Mode',
    editor: 'select',
    options: [
      { label: 'Fast Scan', value: 'fast' },
      { label: 'Thorough Scan', value: 'thorough' },
    ],
  }),
})
```

### Parameter Editors

The `editor` field in `param()` determines how the field is rendered in the UI sidebar:

* `text`: Standard text input.
* `textarea`: Multi-line text area.
* `number`: Numeric input with optional min/max.
* `boolean`: Checkbox/switch.
* `select`: Dropdown menu (requires `options`).
* `multi-select`: Multi-selection dropdown.
* `json`: Code editor for JSON objects.
* `secret`: Masked password-style input.
* `variable-list`: Specialized editor for logic-script variables.

### Visibility Rules

You can use `visibleWhen` to show or hide parameters based on the values of other parameters:

```typescript theme={null}
parameters: parameters({
  useProxy: param(z.boolean().default(false), { editor: 'boolean', label: 'Use Proxy' }),
  proxyUrl: param(z.string().optional(), { 
    editor: 'text', 
    label: 'Proxy URL',
    visibleWhen: { useProxy: true } 
  }),
})
```

### Connection Types

When defining a `port`, you can specify its `connectionType` for compatibility checks in the canvas.

```typescript theme={null}
port(z.string(), {
  label: 'API Key',
  connectionType: { kind: 'primitive', name: 'secret' }
})
```

Supported primitives: `text`, `number`, `boolean`, `secret`, `json`, `file`, `any`.
Lists: `{ kind: 'list', element: ConnectionType }`.
Objects with contracts: `{ kind: 'primitive', name: 'json', contract: 'aws-credentials' }`.

### Entry Point Runtime Input Types

The Entry Point component supports dynamic runtime inputs that users provide when triggering workflows:

| Type     | Description    | UI Rendering                  |
| -------- | -------------- | ----------------------------- |
| `text`   | Text input     | Multi-line textarea           |
| `number` | Numeric input  | Number field                  |
| `file`   | File upload    | File picker                   |
| `json`   | JSON data      | JSON textarea                 |
| `array`  | List of values | Comma-separated or JSON array |
| `secret` | Sensitive data | Password field (masked)       |

**Example: Secret runtime input**

```typescript theme={null}
// Entry point configuration
const runtimeInputs = [
  { id: 'apiKey', label: 'API Key', type: 'secret', required: true },
  { id: 'target', label: 'Target URL', type: 'text', required: true },
];
```

When a workflow with secret inputs is triggered:

1. The UI shows a password field for the secret
2. The value flows through as a `port.secret()` output
3. Downstream components receive the secret string value

***

## Dynamic Ports (resolvePorts)

Components can dynamically generate input/output ports based on parameter values:

```typescript theme={null}
import { inputs, port } from '@shipsec/component-sdk';

export default defineComponent({
  // ...
  parameters: parameters({
    variables: param(z.array(VariableSchema), { ... }),
  }),

  resolvePorts(params) {
    const dynamicInputShape: Record<string, any> = {};
    
    for (const v of params.variables) {
      dynamicInputShape[v.name] = port(z.string(), { 
        label: v.label || v.name 
      });
    }
    
    return {
      inputs: inputs(dynamicInputShape),
    };
  }
});
```

**Use cases:** Workflow calls, Slack templates, manual actions with dynamic options.

***

## Retry Policy

Components can specify custom retry behavior (maps to Temporal activity retry):

```typescript theme={null}
const definition: ComponentDefinition<Input, Output> = {
  id: 'shipsec.api.call',
  // ... other fields ...
  
  retryPolicy: {
    maxAttempts: 5,              // Max retries (0 = unlimited, 1 = no retry)
    initialIntervalSeconds: 2,   // Initial delay
    maximumIntervalSeconds: 120, // Max delay
    backoffCoefficient: 2.0,     // Exponential backoff
    nonRetryableErrorTypes: [    // Errors that should NOT retry
      'AuthenticationError',
      'ValidationError',
    ],
  },
};
```

**Default policy:** 3 attempts, 1s initial, 60s max, 2x backoff.

***

## Error Handling

Use SDK error types for proper retry behavior:

```typescript theme={null}
import { 
  NetworkError,        // Retryable - network issues
  RateLimitError,      // Retryable - with delay
  ServiceError,        // Retryable - 5xx errors
  AuthenticationError, // Non-retryable - bad credentials
  ValidationError,     // Non-retryable - bad input
  NotFoundError,       // Non-retryable - resource missing
  fromHttpResponse,    // Convert HTTP response to error
  wrapError,           // Wrap unknown errors
} from '@shipsec/component-sdk';

async execute({ inputs }, context) {
  try {
    const response = await fetch(inputs.url);
    
    if (!response.ok) {
      throw fromHttpResponse(response, await response.text());
    }
    
    return await response.json();
  } catch (error) {
    throw wrapError(error, 'Failed to call API');
  }
}
```

***

## Analytics Output Port (Results)

Security components should include a `results` output port for analytics integration. This port outputs structured findings that can be indexed into OpenSearch via the Analytics Sink.

### Schema Requirements

The `results` port must output `list<json>` (array of records):

```typescript theme={null}
outputs: outputs({
  // ... other outputs ...

  results: port(z.array(z.record(z.string(), z.unknown())), {
    label: 'Results',
    description:
      'Analytics-ready findings array. Each item includes scanner name and asset key. Connect to Analytics Sink.',
    connectionType: { kind: 'list', element: { kind: 'primitive', name: 'json' } },
  }),
}),
```

### Required Fields

Each finding in the results array **must** include:

| Field          | Type   | Description                                                                 |
| -------------- | ------ | --------------------------------------------------------------------------- |
| `scanner`      | string | Scanner identifier (e.g., `'nuclei'`, `'trufflehog'`, `'supabase-scanner'`) |
| `asset_key`    | string | Primary asset identifier (host, domain, target, etc.)                       |
| `finding_hash` | string | Stable hash for deduplication (16-char hex from SHA-256)                    |

Additional fields from the scanner output should be spread into the finding object.

### Finding Hash

The `finding_hash` is a stable identifier that enables deduplication across workflow runs. It should be generated from the key identifying fields of each finding.

**Purpose:**

* Track if a finding is **new** or **recurring** across scans
* Deduplicate findings in dashboards
* Calculate **first-seen** and **last-seen** timestamps
* Identify which findings have been **resolved** (no longer appearing)

**How to generate:**

Import from the component SDK:

```typescript theme={null}
import { generateFindingHash } from '@shipsec/component-sdk';

// Usage
const hash = generateFindingHash(finding.templateId, finding.host, finding.matchedAt);
```

**Key fields per scanner:**

| Scanner          | Fields Used                          |
| ---------------- | ------------------------------------ |
| Nuclei           | `templateId + host + matchedAt`      |
| TruffleHog       | `DetectorType + Redacted + filePath` |
| Supabase Scanner | `check_id + projectRef + resource`   |

Choose fields that uniquely identify a finding but remain stable across runs (avoid timestamps, random IDs, etc.).

### Example Implementation

```typescript theme={null}
import { generateFindingHash } from '@shipsec/component-sdk';

async execute({ inputs, params }, context) {
  // ... run scanner and get findings ...

  // Build analytics-ready results with scanner metadata
  const results: Record<string, unknown>[] = findings.map((finding) => ({
    ...finding,                           // Spread all finding fields
    scanner: 'my-scanner',               // Scanner identifier
    asset_key: finding.host ?? inputs.target,  // Primary asset
    finding_hash: generateFindingHash(   // Stable deduplication hash
      finding.ruleId,
      finding.host,
      finding.matchedAt
    ),
  }));

  return {
    findings,      // Original findings array
    results,       // Analytics-ready array for Analytics Sink
    rawOutput,     // Raw output for debugging
  };
}
```

### How It Works

1. **Component outputs `results`**: Each scanner outputs its findings with `scanner` and `asset_key` fields
2. **Connect to Analytics Sink**: In the workflow canvas, connect the `results` port to Analytics Sink's `data` input
3. **Indexed to OpenSearch**: Each item in the array becomes a separate document with:
   * Finding data at root level (nested objects serialized to JSON strings)
   * Workflow context under `shipsec.*` namespace
   * Consistent `@timestamp` for all findings in the batch

### Document Structure in OpenSearch

```json theme={null}
{
  "check_id": "DB_RLS_DISABLED",
  "severity": "CRITICAL",
  "title": "RLS Disabled",
  "metadata": "{\"table\":\"users\"}",
  "scanner": "supabase-scanner",
  "asset_key": "abc123xyz",
  "finding_hash": "a1b2c3d4e5f67890",
  "shipsec": {
    "organization_id": "org_123",
    "run_id": "run_abc123",
    "workflow_id": "wf_xyz789",
    "workflow_name": "Supabase Security Audit",
    "component_id": "core.analytics.sink",
    "node_ref": "analytics-sink-1"
  },
  "@timestamp": "2024-01-21T10:30:00Z"
}
```

### `shipsec` Context Fields

The Analytics Sink automatically adds workflow context under the `shipsec` namespace:

| Field             | Description                                   |
| ----------------- | --------------------------------------------- |
| `organization_id` | Organization that owns the workflow           |
| `run_id`          | Unique identifier for this workflow execution |
| `workflow_id`     | ID of the workflow definition                 |
| `workflow_name`   | Human-readable workflow name                  |
| `component_id`    | Component type (e.g., `core.analytics.sink`)  |
| `node_ref`        | Node reference in the workflow graph          |
| `asset_key`       | Auto-detected or specified asset identifier   |

### Example Queries

```
# Find all findings for an asset
asset_key: "api.example.com"

# Find new findings (first seen today)
finding_hash: X AND @timestamp: [now-1d TO now] AND NOT (finding_hash: X AND @timestamp: [* TO now-1d])

# All findings from a specific workflow run
shipsec.run_id: "run_abc123"

# Aggregate findings by scanner
scanner: * | stats count() by scanner

# Track recurring findings
finding_hash: "a1b2c3d4" | sort @timestamp
```

<Note>
  Nested objects in findings are automatically serialized to JSON strings to prevent OpenSearch field explosion (1000 field limit).
</Note>

***

## Docker Component Requirements

<Warning>
  All Docker-based components run with PTY (pseudo-terminal) enabled by default in workflows. Your component MUST be designed for PTY mode.
</Warning>

### Shell Wrapper Pattern (Required)

All Docker-based components MUST use a shell wrapper for PTY compatibility:

```typescript theme={null}
// ✅ CORRECT - Shell wrapper pattern
runner: {
  kind: 'docker',
  image: 'tool:latest',
  entrypoint: 'sh',                      // Shell wrapper
  command: ['-c', 'tool "$@"', '--'],    // Wraps CLI execution
  network: 'bridge',
}
```

```typescript theme={null}
// ❌ WRONG - Direct binary execution
runner: {
  kind: 'docker',
  image: 'tool:latest',
  entrypoint: 'tool',                    // No shell wrapper - will hang
  command: ['-read-stdin', '-output'],
}
```

### Why Shell Wrappers?

| Benefit             | Description                                |
| ------------------- | ------------------------------------------ |
| TTY signal handling | Shell properly handles `SIGTERM`, `SIGHUP` |
| Clean exit          | Shell ensures process cleanup              |
| Buffering control   | Shell manages stdout/stderr correctly      |
| No stdin issues     | Shell doesn't wait for `stdin` input       |

### Pattern Decision Tree

```
Does your Docker image have a shell (/bin/sh)?
├─ YES → Use Shell Wrapper Pattern
│         entrypoint: 'sh', command: ['-c', 'tool "$@"', '--']
│
└─ NO (Distroless) → Use Default Entrypoint Pattern
          Omit entrypoint, pass args via command: []
          The image's built-in ENTRYPOINT handles execution.
```

### Distroless Images (Default Entrypoint Pattern)

Many ProjectDiscovery images (subfinder, dnsx, naabu, amass, notify) are **distroless** and do not contain `/bin/sh`. For these images, omit the `entrypoint` field entirely and let Docker use the image's default entrypoint:

```typescript theme={null}
// ✅ CORRECT - Distroless image (no shell available)
runner: {
  kind: 'docker',
  image: 'ghcr.io/shipsecai/subfinder:latest',
  // No entrypoint — uses image default (/usr/local/bin/subfinder)
  command: [],
  network: 'bridge',
}
```

```typescript theme={null}
// ❌ WRONG - Distroless image with shell wrapper (exit code 127)
runner: {
  kind: 'docker',
  image: 'ghcr.io/shipsecai/subfinder:latest',
  entrypoint: 'sh',  // sh does not exist in distroless images!
  command: ['-c', 'subfinder "$@"', '--'],
}
```

In the `execute()` function, append tool arguments directly to `command`:

```typescript theme={null}
const runnerConfig: DockerRunnerConfig = {
  ...baseRunner,
  command: [...(baseRunner.command ?? []), ...toolArgs],
};
```

<Note>
  Distroless Go binaries (like ProjectDiscovery tools) handle PTY signals correctly.
  Verified with `docker run --rm -t image args...` — output streams and exits cleanly.
</Note>

***

## File System Access

<Warning>
  All components that require file-based input/output MUST use the `IsolatedContainerVolume` utility for multi-tenant security.
</Warning>

For detailed patterns and security guarantees, see [Isolated Volumes](/development/isolated-volumes).

### Quick Example

```typescript theme={null}
import { IsolatedContainerVolume } from '../../utils/isolated-volume';

async execute({ inputs }, context) {
  const tenantId = (context as any).tenantId ?? 'default-tenant';
  const volume = new IsolatedContainerVolume(tenantId, context.runId);

  try {
    await volume.initialize({
      'targets.txt': inputs.targets.join('\n')
    });

    const runnerConfig: DockerRunnerConfig = {
      ...this.runner,
      command: [...(this.runner.command ?? []), '-l', '/inputs/targets.txt'],
      volumes: [volume.getVolumeConfig('/inputs', true)]  // read-only
    };

    return await runComponentWithRunner(runnerConfig, parseOutput, inputs, context);
  } finally {
    await volume.cleanup();  // ALWAYS cleanup
  }
}
```

***

## UI-Only Components

Components that are purely for UI purposes (documentation, notes):

```typescript theme={null}
const definition: ComponentDefinition<Input, void> = {
  id: 'core.ui.text',
  label: 'Text Block',
  category: 'input',
  runner: { kind: 'inline' },
  inputSchema,
  outputSchema: z.void(),
  metadata: {
    uiOnly: true,  // Excluded from workflow execution
  },
  async execute() {
    // No-op for UI-only components
  }
};
```

***

## Testing

### Unit Tests

Located alongside component: `worker/src/components/<category>/__tests__/<component>.test.ts`

```typescript theme={null}
import { describe, it, expect, vi } from 'bun:test';
import * as sdk from '@shipsec/component-sdk';
import { componentRegistry } from '../../index';

describe('my-component', () => {
  it('should process input correctly', async () => {
    const component = componentRegistry.get('my.component.id');
    
    const context = sdk.createExecutionContext({
      runId: 'test-run',
      componentRef: 'test-node',
    });
    
    // Mock the runner for Docker components
    vi.spyOn(sdk, 'runComponentWithRunner').mockResolvedValue('mock output');
    
    const result = await component!.execute({
      inputs: { target: 'example.com' },
      params: {}
    }, context);
    
    expect(result.success).toBe(true);
  });
});
```

**Run:** `bun --cwd worker test`

### Integration Tests (Docker)

Same folder with `-integration.test.ts`. Uses real Docker containers.

```typescript theme={null}
const enableDocker = process.env.ENABLE_DOCKER_TESTS === 'true';
const dockerDescribe = enableDocker ? describe : describe.skip;

dockerDescribe('Component Integration', () => {
  // Tests that run real Docker containers
});
```

**Run:** `ENABLE_DOCKER_TESTS=true bun --cwd worker test`

### Testing Checklist

* [ ] Used `entrypoint: 'sh'` with `command: ['-c', 'tool "$@"', '--']`
* [ ] Tested with `docker run --rm -t` (PTY mode)
* [ ] Container exits cleanly without hanging
* [ ] No stdin-dependent operations
* [ ] Tool arguments appended after `'--'` in command array
* [ ] Workflow run completes successfully

### PTY Testing

```bash theme={null}
# Test with PTY mode (what workflows use)
docker run --rm -t your-image:latest sh -c 'tool "$@"' -- -flag value

# Verify it doesn't wait for stdin
timeout 5 docker run --rm -t your-image:latest sh -c 'tool "$@"' -- --help
```

### E2E Tests (Full Stack)

E2E tests validate your component works with the entire platform: Backend API, Worker, Temporal, and infrastructure.

Located in `e2e-tests/`. These tests create real workflows via the API and execute them.

**Prerequisites:**

```bash theme={null}
# Start full local environment
just dev

# Verify services are running (via nginx)
curl http://localhost/api/v1/health -H "x-internal-token: local-internal-token"
```

**Run E2E tests:**

```bash theme={null}
RUN_E2E=true bun --cwd e2e-tests test
```

**Example E2E test pattern:**

```typescript theme={null}
import { describe, test, expect } from 'bun:test';

const API_BASE = 'http://localhost/api/v1';
const HEADERS = {
  'Content-Type': 'application/json',
  'x-internal-token': 'local-internal-token',
};

// Only run when RUN_E2E=true and services are available
const runE2E = process.env.RUN_E2E === 'true';
const e2eDescribe = runE2E ? describe : describe.skip;

// Helper to poll workflow status until completion
async function pollRunStatus(runId: string, timeoutMs = 180000) {
  const startTime = Date.now();
  while (Date.now() - startTime < timeoutMs) {
    const res = await fetch(`${API_BASE}/workflows/runs/${runId}/status`, { headers: HEADERS });
    const status = await res.json();
    if (['COMPLETED', 'FAILED', 'CANCELLED'].includes(status.status)) {
      return status;
    }
    await new Promise(r => setTimeout(r, 1000));
  }
  throw new Error(`Timeout waiting for workflow ${runId}`);
}

e2eDescribe('My Component E2E', () => {
  test('should execute in a real workflow', async () => {
    // 1. Create workflow with your component
    const workflow = {
      name: 'Test: My Component',
      nodes: [
        {
          id: 'start',
          type: 'core.workflow.entrypoint',
          position: { x: 0, y: 0 },
          data: { label: 'Start', config: { runtimeInputs: [] } },
        },
        {
          id: 'my-node',
          type: 'my.component.id',  // Your component ID
          position: { x: 200, y: 0 },
          data: {
            label: 'My Component',
            config: { target: 'example.com' },
          },
        },
      ],
      edges: [{ id: 'e1', source: 'start', target: 'my-node' }],
    };

    // 2. Create workflow via API
    const createRes = await fetch(`${API_BASE}/workflows`, {
      method: 'POST',
      headers: HEADERS,
      body: JSON.stringify(workflow),
    });
    const { id: workflowId } = await createRes.json();

    // 3. Execute workflow
    const runRes = await fetch(`${API_BASE}/workflows/${workflowId}/run`, {
      method: 'POST',
      headers: HEADERS,
      body: JSON.stringify({ inputs: {} }),
    });
    const { runId } = await runRes.json();

    // 4. Poll until completion
    const result = await pollRunStatus(runId);
    
    // 5. Assert results
    expect(result.status).toBe('COMPLETED');
  }, 180000);  // 3 minute timeout for workflow execution
});
```

<Note>
  E2E tests are **not** run in CI yet. They require the full local environment (`just dev`) and are intended for manual validation during development.
</Note>

## Complete Example

```typescript theme={null}
import { z } from 'zod';
import { 
  componentRegistry, 
  ComponentDefinition,
  DockerRunnerConfig,
  runComponentWithRunner,
  port,
} from '@shipsec/component-sdk';
import { IsolatedContainerVolume } from '../../utils/isolated-volume';

const inputSchema = z.object({
  domains: z.array(z.string()).min(1),
  threads: z.number().optional().default(10),
});

const outputSchema = z.object({
  results: z.array(z.object({
    domain: z.string(),
    records: z.array(z.string()),
  })),
  rawOutput: z.string(),
});

type Input = z.infer<typeof inputSchema>;
type Output = z.infer<typeof outputSchema>;

export default defineComponent({
  id: 'shipsec.dnsx.scan',
  label: 'DNSX Scanner',
  category: 'security',
  runner: {
    kind: 'docker',
    image: 'ghcr.io/shipsecai/dnsx:latest',
    entrypoint: 'sh',
    command: ['-c', 'dnsx "$@"', '--'],
    network: 'bridge',
  },
  
  inputs: inputs({
    domains: port(z.array(z.string()).min(1), { 
      label: 'Domains',
      connectionType: { kind: 'list', element: { kind: 'primitive', name: 'text' } }
    }),
  }),

  parameters: parameters({
    threads: param(z.number().default(10), {
      label: 'Threads',
      editor: 'number',
      min: 1,
      max: 100,
    }),
  }),

  outputs: outputs({
    results: port(z.array(z.object({
      domain: z.string(),
      records: z.array(z.string()),
    })), { label: 'Scan Results' }),
    rawOutput: port(z.string(), { label: 'Raw Stdout' }),
  }),
  
  async execute({ inputs, params }, context) {
    const tenantId = (context as any).tenantId ?? 'default-tenant';
    const volume = new IsolatedContainerVolume(tenantId, context.runId);

    try {
      context.emitProgress('Preparing input files...');
      await volume.initialize({
        'domains.txt': inputs.domains.join('\n')
      });

      const args = [
        '-l', '/inputs/domains.txt',
        '-json',
        '-t', String(params.threads),
        '-stream',
      ];

      const runnerConfig: DockerRunnerConfig = {
        ...this.runner,
        command: [...(this.runner.command ?? []), ...args],
        volumes: [volume.getVolumeConfig('/inputs', true)],
      };

      context.emitProgress('Running DNSX...');
      const rawOutput = await runComponentWithRunner(
        runnerConfig,
        async (stdout) => {
          const lines = stdout.split('\n').filter(Boolean);
          const results = lines.map(line => {
            const parsed = JSON.parse(line);
            return {
              domain: parsed.host,
              records: parsed.a || [],
            };
          });
          return { results, rawOutput: stdout };
        },
        inputs,
        context
      );

      context.logger.info(`Found ${rawOutput.results.length} results`);
      return rawOutput;

    } finally {
      await volume.cleanup();
    }
  }
});

componentRegistry.register(definition);
export default definition;
```

***

## Questions?

* File access patterns: See [Isolated Volumes](/development/isolated-volumes)
* SDK source: `packages/component-sdk/src/`
* Example components: `worker/src/components/security/`
* Bug reports: GitHub Issues


# Isolated Volumes
Source: https://docs.shipsec.ai/development/isolated-volumes

Docker-in-Docker file mounting solution for multi-tenant security

## Problem Summary

In a Docker-in-Docker (DinD) setup where the worker container runs inside Docker and creates containers:

1. **Volume Path Mismatch**: Volume mount paths are relative to the Docker daemon's filesystem, not the worker container
2. **Security Risk**: Shared volumes in multi-tenant SaaS allow cross-tenant data access
3. **Limited stdin Approach**: Using stdin for data transfer doesn't support file-based tools

***

## Solution: Isolated Named Volumes

Use unique Docker named volumes created per `tenantId + runId + timestamp`:

```
tenant-${tenantId}-run-${runId}-${timestamp}
```

### Architecture

```
┌─────────────────────────────────────────────────────────┐
│ Docker Host                                              │
│                                                          │
│  ┌─────────────────────────────────────────────┐        │
│  │ Worker Container (DinD)                     │        │
│  │                                              │        │
│  │  1. Creates volume via Docker CLI           │        │
│  │     docker volume create tenant-A-run-1-... │        │
│  │                                              │        │
│  │  2. Populates files using temp container    │        │
│  │     docker run -v vol:/data alpine sh -c .. │        │
│  │                                              │        │
│  │  3. Runs actual tool with volume mounted    │        │
│  │     docker run -v vol:/inputs dnsx ...      │        │
│  │                                              │        │
│  │  4. Reads output files using temp container │        │
│  │     docker run -v vol:/data alpine cat ...  │        │
│  │                                              │        │
│  │  5. Cleans up volume                        │        │
│  │     docker volume rm tenant-A-run-1-...     │        │
│  └─────────────────────────────────────────────┘        │
│                                                          │
│  ┌──────────────────────────────────────────┐           │
│  │ Docker Volumes (on Docker Host)          │           │
│  │                                           │           │
│  │  • tenant-A-run-123-1732090000           │           │
│  │  • tenant-B-run-456-1732090001           │           │
│  │  • tenant-A-run-789-1732090002           │           │
│  │                                           │           │
│  │  Each volume isolated per tenant + run   │           │
│  └──────────────────────────────────────────┘           │
└─────────────────────────────────────────────────────────┘
```

***

## Security Benefits

| Aspect               | Old Approach                    | Isolated Volumes                 |
| -------------------- | ------------------------------- | -------------------------------- |
| **Tenant Isolation** | ❌ Shared volume or stdin        | ✅ Unique volume per tenant+run   |
| **Path Traversal**   | ⚠️ Possible with file mounts    | ✅ Validated filenames            |
| **Data Leakage**     | ❌ Files persist in shared space | ✅ Immediate cleanup              |
| **Audit Trail**      | ❌ None                          | ✅ Volume labels track tenant/run |
| **DinD Compatible**  | ❌ File mounts don't work        | ✅ Named volumes work perfectly   |

***

## Implementation

### Before: File Mounting (Broken in DinD)

```typescript theme={null}
// ❌ WRONG - Breaks in DinD, no tenant isolation
const hostInputDir = await mkdtemp(path.join(tmpdir(), 'dnsx-input-'));
await writeFile(path.join(hostInputDir, 'file.txt'), data);

const runnerConfig: DockerRunnerConfig = {
  volumes: [
    { source: hostInputDir, target: '/inputs', readOnly: true }
  ]
};
```

### After: Isolated Volumes (DinD Compatible)

```typescript theme={null}
// ✅ CORRECT - DinD compatible, tenant isolated
const tenantId = context.tenantId ?? 'default-tenant';
const volume = new IsolatedContainerVolume(tenantId, context.runId);

try {
  await volume.initialize({
    'domains.txt': domains.join('\n'),
    'resolvers.txt': resolvers.join('\n')
  });

  const runnerConfig: DockerRunnerConfig = {
    volumes: [volume.getVolumeConfig('/inputs', true)]
  };

  await runComponentWithRunner(runnerConfig, ...);

  const outputs = await volume.readFiles(['results.json']);

} finally {
  await volume.cleanup();
}
```

***

## Comparison: All Approaches

| Feature              | File Mounts    | stdin Approach   | Isolated Volumes |
| -------------------- | -------------- | ---------------- | ---------------- |
| **DinD Compatible**  | ❌ No           | ✅ Yes            | ✅ Yes            |
| **File-based tools** | ✅ Yes          | ❌ No             | ✅ Yes            |
| **Config files**     | ✅ Yes          | ❌ No             | ✅ Yes            |
| **Output files**     | ❌ Hard to read | ❌ No             | ✅ Yes            |
| **Binary files**     | ✅ Yes          | ❌ No             | ✅ Yes            |
| **Large files**      | ✅ Yes          | ⚠️ Memory limits | ✅ Yes            |
| **Tenant isolation** | ❌ No           | ⚠️ Process-level | ✅ Volume-level   |

***

## Usage Examples

### Simple Input Files

```typescript theme={null}
const volume = new IsolatedContainerVolume(tenantId, runId);

try {
  await volume.initialize({
    'targets.txt': targets.join('\n')
  });

  const config = {
    volumes: [volume.getVolumeConfig('/inputs', true)]
  };

  await runTool(config);
} finally {
  await volume.cleanup();
}
```

### Input + Output Files

```typescript theme={null}
const volume = new IsolatedContainerVolume(tenantId, runId);

try {
  await volume.initialize({
    'config.yaml': yamlConfig
  });

  const config = {
    command: ['--input', '/data/config.yaml', '--output', '/data/results.json'],
    volumes: [volume.getVolumeConfig('/data', false)] // Read-write
  };

  await runTool(config);

  const outputs = await volume.readFiles(['results.json', 'summary.txt']);
  return JSON.parse(outputs['results.json']);

} finally {
  await volume.cleanup();
}
```

### Multiple Volumes

```typescript theme={null}
const inputVol = new IsolatedContainerVolume(tenantId, `${runId}-in`);
const outputVol = new IsolatedContainerVolume(tenantId, `${runId}-out`);

try {
  await inputVol.initialize({ 'data.csv': csvData });
  await outputVol.initialize({}); // Empty volume for outputs

  const config = {
    volumes: [
      inputVol.getVolumeConfig('/inputs', true),
      outputVol.getVolumeConfig('/outputs', false)
    ]
  };

  await runTool(config);

  const results = await outputVol.readFiles(['output.json']);

} finally {
  await Promise.all([
    inputVol.cleanup(),
    outputVol.cleanup()
  ]);
}
```

***

## Volume Lifecycle

1. **Create**: `docker volume create tenant-A-run-123-...`
2. **Populate**: Use temporary Alpine container to write files
3. **Mount**: Container uses the volume via `-v volumeName:/path`
4. **Read**: Use temporary Alpine container to read files
5. **Cleanup**: `docker volume rm tenant-A-run-123-...`

### Automatic Cleanup

Volumes are always cleaned up via `finally` blocks:

```typescript theme={null}
try {
  await volume.initialize(...);
  await runTool(...);
} finally {
  await volume.cleanup(); // Always runs, even on error
}
```

### Orphan Cleanup

For volumes that weren't cleaned up (e.g., worker crash):

```bash theme={null}
# List studio-managed volumes
docker volume ls --filter "label=studio.managed=true"

# Remove old volumes
docker volume prune --filter "label=studio.managed=true"
```

***

## Security Requirements

### Tenant Isolation

Every execution gets a unique volume:

```
tenant-{tenantId}-run-{runId}-{timestamp}
```

Example: `tenant-acme-run-wf-abc123-1732150000`

### Read-Only Mounts

```typescript theme={null}
// Input files should be read-only
volume.getVolumeConfig('/inputs', true)  // ✅ read-only

// Only make writable if tool needs to write
volume.getVolumeConfig('/outputs', false)  // ⚠️ read-write
```

### Nonroot Container Support

Volumes automatically support containers running as nonroot users (e.g., distroless images with uid 65532).

After files are written to the volume, permissions are set to `777` to allow any container user to read/write:

```typescript theme={null}
// This happens automatically in volume.initialize()
// chmod -R 777 /data
```

**Why this is needed:**

* Files are written to volumes using Alpine containers (running as root)
* Distroless nonroot images run as uid 65532
* Without permission changes, nonroot containers can't write output files

**This is safe because:**

* Each volume is isolated per tenant + run
* Volumes are cleaned up after execution
* No cross-tenant access is possible

### Path Validation

Filenames are automatically validated:

```typescript theme={null}
// ✅ OK
await volume.initialize({
  'file.txt': data,
  'subdir/file.txt': data  // Subdirs OK
});

// ❌ Rejected (security)
await volume.initialize({
  '../file.txt': data,     // Path traversal blocked
  '/etc/passwd': data      // Absolute paths blocked
});
```

***

## Security Guarantees

| Security Feature      | How It Works                          |
| --------------------- | ------------------------------------- |
| **Tenant Isolation**  | Volume name includes tenant ID        |
| **No Collisions**     | Timestamp prevents conflicts          |
| **Path Safety**       | Filenames validated (no `..` or `/`)  |
| **Automatic Cleanup** | Finally blocks guarantee removal      |
| **Audit Trail**       | Volumes labeled `studio.managed=true` |
| **DinD Compatible**   | Named volumes work in nested Docker   |

***

## Performance

### Volume Creation Overhead

* **Creation**: \~50-100ms per volume
* **File writes**: \~10-50ms per file (depends on size)
* **Cleanup**: \~50-100ms per volume

**Total overhead**: \~100-250ms per execution

This is acceptable for security tools that typically run for seconds/minutes.

### Optimization Tips

1. **Batch file writes**: Write all files in one `initialize()` call
2. **Reuse volumes**: For sequential operations in same run, reuse the volume
3. **Lazy cleanup**: Clean up volumes in background job if latency-sensitive

***

## When to Use Each Approach

### Use Isolated Volumes When:

* ✅ Running in DinD environment
* ✅ Need multi-tenant isolation
* ✅ Tool requires file-based config
* ✅ Tool writes output files
* ✅ Handling binary/large files

### Use stdin/stdout When:

* ✅ Tool supports stdin input
* ✅ Single-tenant or dev environment
* ✅ Small text-only inputs
* ✅ Don't need output files

### Use File Mounts When:

* ✅ NOT running in DinD (direct Docker)
* ✅ Development/testing only
* ✅ Quick prototyping

***

## Migration Checklist

To migrate a component to use isolated volumes:

* [ ] Import `IsolatedContainerVolume`
* [ ] Get `tenantId` from context (use fallback for now)
* [ ] Create volume instance: `new IsolatedContainerVolume(tenantId, runId)`
* [ ] Replace file writes with `volume.initialize({ files })`
* [ ] Replace volume mount with `volume.getVolumeConfig()`
* [ ] Add `finally` block with `volume.cleanup()`
* [ ] If tool writes outputs, use `volume.readFiles()` to retrieve them
* [ ] Test in DinD environment


# Release Process
Source: https://docs.shipsec.ai/development/release-process

Docker image releases and npm package versioning

This document describes the release process for ShipSec Studio, including Docker image releases and npm package versioning.

***

## Docker Image Releases

### Automatic Release (Recommended)

When you push a git tag matching the pattern `v*.*.*` (e.g., `v1.0.0`, `v1.2.3`), the GitHub Actions workflow automatically:

1. **Builds** all three Docker images (backend, worker, frontend)
2. **Tags** images with:
   * Version tag: `ghcr.io/shipsecai/studio-{service}:{version}`
   * Latest tag: `ghcr.io/shipsecai/studio-{service}:latest`
3. **Pushes** images to GitHub Container Registry (GHCR)
4. **Creates** a GitHub release with changelog

### Creating a Release

```bash theme={null}
# 1. Ensure you're on main branch and up to date
git checkout main
git pull origin main

# 2. Create and push a tag
git tag -a v1.0.0 -m "Release v1.0.0"
git push origin v1.0.0

# The workflow will automatically build and release
```

### Manual Release

You can also trigger a release manually via GitHub Actions UI:

1. Go to Actions → Release workflow
2. Click "Run workflow"
3. Enter the version tag (e.g., `v1.0.0`)

***

## Image Tags

Images are tagged as:

| Service  | Image                                         |
| -------- | --------------------------------------------- |
| Backend  | `ghcr.io/shipsecai/studio-backend:{version}`  |
| Worker   | `ghcr.io/shipsecai/studio-worker:{version}`   |
| Frontend | `ghcr.io/shipsecai/studio-frontend:{version}` |

All images also have a `:latest` tag.

### Pulling Images

```bash theme={null}
# Pull a specific version
docker pull ghcr.io/shipsecai/studio-backend:1.0.0

# Pull latest
docker pull ghcr.io/shipsecai/studio-backend:latest
```

***

## NPM Package Versioning

### Current State

This is a **monorepo** with workspace packages:

| Package                    | Version | Published    |
| -------------------------- | ------- | ------------ |
| `@shipsec/shared`          | v0.1.0  | No           |
| `@shipsec/component-sdk`   | v0.1.0  | No (private) |
| `@shipsec/backend-client`  | v0.1.0  | No           |
| `@shipsec/studio-backend`  | v0.1.0  | No           |
| `@shipsec/studio-worker`   | v0.1.0  | No (private) |
| `@shipsec/studio-frontend` | v1.0.0  | No           |

<Note>
  These packages are **workspace dependencies** and are **not published to npm**. They're consumed internally within the monorepo.
</Note>

### Versioning Options

#### Option 1: Manual Versioning (Current)

**When to use**: Small teams, infrequent releases, packages not published to npm.

```bash theme={null}
# Update package.json versions manually before release
# Commit version changes
# Tag release
```

#### Option 2: Sync with Docker Release Tag (Recommended)

Since packages aren't published to npm, sync versions with Docker releases:

```bash theme={null}
# Sync all package versions with release tag
bun run scripts/sync-versions.ts v1.0.0
```

#### Option 3: Changesets (For Published Packages)

**When to use**: If you plan to publish packages to npm.

```bash theme={null}
bun add -D @changesets/cli
bun changeset init
bun changeset          # Add changeset
bun changeset version  # Version packages
bun changeset publish  # Publish to npm
```

***

## Release Checklist

Before creating a release:

* [ ] Update CHANGELOG.md (if maintained)
* [ ] Run tests: `bun run test`
* [ ] Run typecheck: `bun run typecheck`
* [ ] Run lint: `bun run lint`
* [ ] Update package versions (if syncing): `bun run scripts/sync-versions.ts v1.0.0`
* [ ] Commit version changes (if any)

Create the release:

* [ ] Create and push tag: `git tag -a v1.0.0 -m "Release v1.0.0" && git push origin v1.0.0`
* [ ] Verify GitHub Actions workflow completes
* [ ] Verify images are available in GHCR
* [ ] Verify GitHub release is created

***

## Required GitHub Secrets

The release workflow uses `GITHUB_TOKEN` which is automatically provided by GitHub Actions. No additional secrets are required for GHCR.

***

## Troubleshooting

### Images not appearing in GHCR

1. Check workflow logs for errors
2. Verify `GITHUB_TOKEN` has `packages:write` permission
3. Ensure repository has GitHub Packages enabled

### Release not created

1. Check workflow logs
2. Verify tag format matches `v*.*.*`
3. Check GitHub Actions permissions

### Build failures

1. Run tests locally: `bun run test`
2. Run typecheck: `bun run typecheck`
3. Check for uncommitted changes
4. Verify all dependencies are installed

***

## Semantic Versioning

Follow [Semantic Versioning](https://semver.org/):

| Change Type      | Version Bump | Example           |
| ---------------- | ------------ | ----------------- |
| Breaking changes | Major        | `1.0.0` → `2.0.0` |
| New features     | Minor        | `1.0.0` → `1.1.0` |
| Bug fixes        | Patch        | `1.0.0` → `1.0.1` |

### Commit Message Convention

Use [Conventional Commits](https://www.conventionalcommits.org/):

```
feat: add new workflow component
fix: resolve authentication issue
docs: update installation guide
chore: update dependencies
```


# Workflow Analytics
Source: https://docs.shipsec.ai/development/workflow-analytics

Index security findings into OpenSearch for dashboards, queries, and alerting

ShipSec Studio includes a workflow analytics system that indexes security findings into OpenSearch. This enables real-time dashboards, historical trend analysis, and alerting on security data.

***

## Overview

The analytics system consists of:

1. **Analytics Sink component** - Indexes workflow output data into OpenSearch
2. **Results output port** - Structured findings from security scanners
3. **OpenSearch storage** - Time-series index for querying and visualization
4. **View Analytics button** - Quick access to filtered dashboards

***

## Architecture

```mermaid theme={null}
flowchart LR
    subgraph Scanners
        N[Nuclei Scan]
        T[TruffleHog]
        S[Supabase Scanner]
    end

    AS[Analytics Sink]
    OS[(OpenSearch)]

    subgraph Dashboard[OpenSearch Dashboards]
        V[Visualizations]
        A[Alerts]
        Q[Queries]
    end

    N -->|results port| AS
    T -->|results port| AS
    S -->|results port| AS
    AS --> OS
    OS --> Dashboard
```

Each scanner outputs findings through its `results` port, which connects to the Analytics Sink. The sink indexes each finding as a separate document with workflow metadata.

***

## Document Structure

Indexed documents follow this structure:

```json theme={null}
{
  "check_id": "DB_RLS_DISABLED",
  "severity": "CRITICAL",
  "title": "RLS Disabled on Table: users",
  "resource": "public.users",
  "metadata": "{\"schema\":\"public\",\"table\":\"users\"}",
  "scanner": "supabase-scanner",
  "asset_key": "abcdefghij1234567890",
  "finding_hash": "a1b2c3d4e5f67890",
  "shipsec": {
    "organization_id": "org_123",
    "run_id": "shipsec-run-xxx",
    "workflow_id": "d1d33161-929f-4af4-9a64-xxx",
    "workflow_name": "Supabase Security Audit",
    "component_id": "core.analytics.sink",
    "node_ref": "analytics-sink-1"
  },
  "@timestamp": "2025-01-21T10:30:00.000Z"
}
```

### Field Categories

| Category         | Fields                                 | Description                           |
| ---------------- | -------------------------------------- | ------------------------------------- |
| Finding data     | `check_id`, `severity`, `title`, etc.  | Scanner-specific fields at root level |
| Asset tracking   | `scanner`, `asset_key`, `finding_hash` | Required fields for analytics         |
| Workflow context | `shipsec.*`                            | Automatic metadata from the workflow  |
| Timestamp        | `@timestamp`                           | Indexing timestamp                    |

<Note>
  Nested objects in findings are automatically serialized to JSON strings to prevent OpenSearch field explosion (1000 field limit).
</Note>

***

## `shipsec` context fields

The Analytics Sink automatically adds workflow context under the `shipsec` namespace:

| Field             | Description                                   |
| ----------------- | --------------------------------------------- |
| `organization_id` | Organization that owns the workflow           |
| `run_id`          | Unique identifier for this workflow execution |
| `workflow_id`     | ID of the workflow definition                 |
| `workflow_name`   | Human-readable workflow name                  |
| `component_id`    | Component type (always `core.analytics.sink`) |
| `node_ref`        | Node reference in the workflow graph          |
| `asset_key`       | Auto-detected or specified asset identifier   |

***

## Finding Hash for Deduplication

The `finding_hash` is a stable 16-character identifier that enables tracking findings across workflow runs.

### Purpose

* **New vs recurring**: Determine if a finding appeared before
* **First-seen / last-seen**: Track when findings were first and last detected
* **Resolution tracking**: Findings that stop appearing may be resolved
* **Deduplication**: Remove duplicates in dashboards across runs

### Generation

Each scanner generates the hash from key identifying fields:

| Scanner          | Hash Fields                          |
| ---------------- | ------------------------------------ |
| Nuclei           | `templateId + host + matchedAt`      |
| TruffleHog       | `DetectorType + Redacted + filePath` |
| Supabase Scanner | `check_id + projectRef + resource`   |

Fields are normalized (lowercase, trimmed) and hashed with SHA-256, truncated to 16 hex characters.

***

## Querying Data

### Basic Queries (KQL)

```
# Find all findings for an asset
asset_key: "api.example.com"

# Filter by severity
severity: "CRITICAL" OR severity: "HIGH"

# Filter by scanner
scanner: "nuclei"

# All findings from a specific workflow run
shipsec.run_id: "shipsec-run-abc123"

# Filter by organization
shipsec.organization_id: "org_123"

# Filter by workflow
shipsec.workflow_id: "d1d33161-929f-4af4-9a64-xxx"
```

### Tracking Findings Over Time

```
# Track a specific finding across runs
finding_hash: "a1b2c3d4e5f67890"

# Find recurring findings (multiple runs)
# Use aggregation: group by finding_hash, count occurrences
```

### Common Aggregations

| Aggregation                      | Use Case                   |
| -------------------------------- | -------------------------- |
| `terms` on `severity`            | Count findings by severity |
| `terms` on `scanner`             | Count findings by scanner  |
| `terms` on `asset_key`           | Most vulnerable assets     |
| `date_histogram` on `@timestamp` | Findings over time         |
| `cardinality` on `finding_hash`  | Unique findings count      |

***

## Setting Up OpenSearch

### Environment Variables

Set these in your `worker/.env`:

```bash theme={null}
OPENSEARCH_URL=http://localhost:9200
OPENSEARCH_USERNAME=admin
OPENSEARCH_PASSWORD=admin
```

Set this in your `frontend/.env`:

```bash theme={null}
VITE_OPENSEARCH_DASHBOARDS_URL=http://localhost/analytics
```

### Docker Compose

The infrastructure stack includes OpenSearch and OpenSearch Dashboards:

```bash theme={null}
docker compose -f docker/docker-compose.infra.yml up -d opensearch opensearch-dashboards
```

### Index Pattern

After indexing data, create an index pattern in OpenSearch Dashboards:

1. Go to **Dashboards Management** > **Index Patterns**
2. Create pattern: `security-findings-*`
3. Select `@timestamp` as the time field
4. Click **Create index pattern**

<Warning>
  If you don't see `shipsec.*` fields in Available Fields after indexing, refresh the index pattern field list in Dashboards Management.
</Warning>

***

## Analytics API Limits

The analytics query API enforces sane bounds to protect OpenSearch:

* `size` must be a non-negative integer and is capped at **1000**
* `from` must be a non-negative integer and is capped at **10000**

Requests above these limits return `400 Bad Request`.

## Analytics Settings Updates

The analytics settings API supports partial updates:

* `analyticsRetentionDays` is optional
* `subscriptionTier` is optional

Omit fields you don’t want to change. The backend validates retention days only when provided.

***

## Using Analytics Sink

### Basic Workflow

1. Add a security scanner to your workflow (Nuclei, TruffleHog, etc.)
2. Add an Analytics Sink component
3. Connect the scanner's `results` port to the Analytics Sink's `data` input
4. Run the workflow

### Component Parameters

| Parameter             | Description                                                                                                              |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **Index Suffix**      | Custom suffix for the index name. Defaults to slugified workflow name.                                                   |
| **Asset Key Field**   | Field to use as asset identifier. Auto-detect checks: asset\_key > host > domain > subdomain > url > ip > asset > target |
| **Custom Field Name** | Custom field when Asset Key Field is "custom"                                                                            |
| **Fail on Error**     | When enabled, workflow stops if indexing fails. Default: fire-and-forget.                                                |

### Fire-and-Forget Mode

By default, Analytics Sink operates in fire-and-forget mode:

* Indexing errors are logged but don't stop the workflow
* Useful for non-critical analytics that shouldn't block security scans
* Enable "Fail on Error" for strict indexing requirements

***

## View Analytics Button

The workflow builder includes a "View Analytics" button that opens OpenSearch Dashboards with pre-filtered data:

* **When a run is selected**: Filters by `shipsec.run_id`
* **When no run is selected**: Filters by `shipsec.workflow_id`
* **Time range**: Last 7 days

The button only appears when `VITE_OPENSEARCH_DASHBOARDS_URL` is configured.

***

## Index Naming

Indexes follow the pattern: `security-findings-{orgId}-{suffix}`

| Component | Value                                           |
| --------- | ----------------------------------------------- |
| `orgId`   | Organization ID from workflow context           |
| `suffix`  | Custom suffix parameter, or date (`YYYY.MM.DD`) |

Example: `security-findings-org_abc123-2025.01.21`

***

## Building Dashboards

### Recommended Visualizations

| Visualization             | Description                                             |
| ------------------------- | ------------------------------------------------------- |
| **Findings Over Time**    | Line chart with `@timestamp` on X-axis, count on Y-axis |
| **Severity Distribution** | Pie chart with `terms` on `severity`                    |
| **Top Vulnerable Assets** | Bar chart with `terms` on `asset_key`                   |
| **Findings by Scanner**   | Bar chart with `terms` on `scanner`                     |
| **New vs Recurring**      | Use `finding_hash` cardinality vs total count           |

### Alert Examples

| Alert                     | Query                         |
| ------------------------- | ----------------------------- |
| Critical finding detected | `severity: "CRITICAL"`        |
| New secrets exposed       | `scanner: "trufflehog"`       |
| RLS disabled              | `check_id: "DB_RLS_DISABLED"` |

***

## Troubleshooting

### Data not appearing in OpenSearch

1. Check worker logs for `[OpenSearchIndexer]` messages
2. Verify `OPENSEARCH_URL` is set in worker environment
3. Ensure Analytics Sink is connected to a `results` port
4. Check if OpenSearch is running: `curl http://localhost:9200/_cluster/health`

### Field mapping errors

If you see "Limit of total fields \[1000] has been exceeded":

1. Delete the problematic index: `curl -X DELETE "http://localhost:9200/security-findings-*"`
2. Re-run the workflow (new index will use correct schema)

### shipsec fields not visible

1. Fields starting with `_` are hidden in OpenSearch UI
2. Ensure you're using `shipsec.*` (no underscore prefix)
3. Refresh the index pattern in Dashboards Management

### pm2 not loading environment variables

pm2's `env_file` doesn't auto-inject variables. The worker uses a custom `loadWorkerEnv()` function in `pm2.config.cjs`. After changing `worker/.env`:

```bash theme={null}
pm2 delete shipsec-worker
pm2 start pm2.config.cjs --only shipsec-worker
```

***

## Best Practices

### Do

* Connect `results` ports (not `rawOutput`) to Analytics Sink
* Use meaningful index suffixes for organization
* Monitor index size and implement retention policies
* Create saved searches for common queries

### Don't

* Don't connect deeply nested JSON (causes field explosion)
* Don't rely on analytics for critical workflow logic
* Don't store PII or secrets in indexed findings

***

## Component Author Guidelines

If you're building a security scanner component, see [Analytics Output Port](/development/component-development#analytics-output-port-results) for implementation details on adding the `results` output port.

***

## Related

* [Component Development](/development/component-development) - Building scanner components
* [Core Components](/components/core) - Analytics Sink reference
* [Analytics (PostHog)](/development/analytics) - Product analytics (different system)


# Introduction
Source: https://docs.shipsec.ai/index

Open source, no-code security automation platform for security teams

**ShipSec Studio** is the open source, no-code security automation platform for security teams. Design reusable security workflows, see every run unfold live, and keep humans in the loop with opinionated guardrails.

<Frame>
  <img alt="Workflow Builder" />
</Frame>

## Why ShipSec Studio?

* **No code required** – Build powerful security workflows by connecting components visually. No scripting, no glue code, no maintenance headaches.
* **Built for security teams** – Pre-built components for subdomain discovery, vulnerability scanning, secret detection, and more. Focus on security, not infrastructure.
* **Workflows that don't fail silently** – Automatic retries, resumable executions, and real-time log streaming, giving visibility into every step. Know exactly what's happening and why.
* **Self-hosted and open source** – Your data stays on your infrastructure. Run it on a laptop or scale to production with the same codebase.
* **Secure by default** – Encrypted secrets, isolated container execution, and team-based access controls built in from day one.

## Product Tour

<CardGroup>
  <Card title="Workflow Builder" icon="diagram-project">
    Visual drag-and-drop editor for building security workflows

    <Frame>
      <img alt="Workflow Builder" />
    </Frame>
  </Card>

  <Card title="Live Execution" icon="play">
    Watch workflow execution unfold in real-time with streaming logs

    <Frame>
      <img alt="Execution Timeline" />
    </Frame>
  </Card>

  <Card title="Secrets Management" icon="key">
    Secure storage for API keys and credentials

    <Frame>
      <img alt="Secret Manager" />
    </Frame>
  </Card>

  <Card title="Connections" icon="plug">
    Integrate with external services and APIs

    <Frame>
      <img alt="Connections Catalog" />
    </Frame>
  </Card>
</CardGroup>

## Get Started

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="/quickstart">
    Get up and running in 5 minutes
  </Card>

  <Card title="Installation" icon="download" href="/installation">
    Self-host on your own infrastructure
  </Card>

  <Card title="Architecture" icon="sitemap" href="/architecture">
    Understand the system design
  </Card>

  <Card title="Components" icon="cube" href="/components/overview">
    Explore available components
  </Card>
</CardGroup>

## Technology Stack

| Layer              | Technologies                                                            |
| ------------------ | ----------------------------------------------------------------------- |
| **Frontend**       | React 19, TypeScript, Vite, Tailwind CSS, Radix UI, ReactFlow, xterm.js |
| **Backend**        | NestJS, TypeScript, Bun runtime, PostgreSQL, Drizzle ORM, Clerk Auth    |
| **Worker**         | Node.js, TypeScript, Temporal.io, Docker containers                     |
| **Infrastructure** | PostgreSQL, Temporal, MinIO, Redis, Loki, Redpanda (Kafka)              |

## Need Help?

* **Support** – [support@shipsec.ai](mailto:support@shipsec.ai)
* **GitHub** – [github.com/shipsecai/studio](https://github.com/shipsecai/studio)


# Installation
Source: https://docs.shipsec.ai/installation

Self-host ShipSec Studio on your infrastructure

## Prerequisites

* Docker Desktop with ≥8 GB RAM
* [Bun](https://bun.sh) runtime
* [just](https://github.com/casey/just) command runner
* Open ports: `5433`, `7233`, `8081`, `9000`, `9001`, `3100`

***

## Installation Steps

### 1. Clone & initialize

```bash theme={null}
git clone https://github.com/shipsecai/studio.git
cd studio
just init
```

<Note>
  **macOS users:** If you see native module errors, run:

  ```bash theme={null}
  export SDKROOT=$(xcrun --show-sdk-path)
  bun install
  ```
</Note>

The `just init` command automatically:

* Installs project dependencies with Bun
* Creates environment files from examples (`backend/.env`, `worker/.env`, `frontend/.env`)

### 2. Configure environment variables

Edit the generated `.env` files as needed:

```bash theme={null}
# Edit environment files
nano backend/.env
nano worker/.env
nano frontend/.env
```

### 3. Start development environment

```bash theme={null}
just dev
```

The `just dev` command automatically:

* Checks that environment files exist
* Starts Docker infrastructure (Postgres, Temporal, MinIO, Loki)
* Waits for services to be ready
* Runs database migrations
* Starts backend, worker, and frontend with hot-reload

***

## Service Endpoints

In development mode, all application services are accessible through nginx on port 80:

| Service       | URL                                                        | Notes                                |
| ------------- | ---------------------------------------------------------- | ------------------------------------ |
| Frontend      | [http://localhost/](http://localhost/)                     | Workflow builder (via nginx)         |
| Backend API   | [http://localhost/api/](http://localhost/api/)             | REST API (via nginx)                 |
| Analytics     | [http://localhost/analytics/](http://localhost/analytics/) | OpenSearch Dashboards (via nginx)    |
| Temporal UI   | [http://localhost:8081](http://localhost:8081)             | Workflow management                  |
| PostgreSQL    | [http://localhost:5433](http://localhost:5433)             | Database                             |
| MinIO         | [http://localhost:9000](http://localhost:9000)             | Object storage                       |
| MinIO Console | [http://localhost:9001](http://localhost:9001)             | Credentials: `minioadmin/minioadmin` |
| Loki          | [http://localhost:3100](http://localhost:3100)             | Logs                                 |

<Note>
  Individual service ports (5173, 3211, 5601) are available for debugging but should not be used in normal development. All traffic flows through nginx on port 80.
</Note>

***

## Analytics Stack (Optional)

ShipSec Studio includes an optional analytics stack powered by OpenSearch for indexing and visualizing workflow execution data.

### Starting the Analytics Stack

The analytics services are included in the infrastructure docker-compose file:

```bash theme={null}
# Start infrastructure including OpenSearch
just infra up
```

This will start:

* **OpenSearch** on port `9200` - Search and analytics engine
* **OpenSearch Dashboards** on port `5601` - Visualization and query UI

### Configuring Analytics

Add these environment variables to your backend and worker `.env` files:

```bash theme={null}
# Backend (.env)
OPENSEARCH_URL=http://localhost:9200
OPENSEARCH_USERNAME=admin
OPENSEARCH_PASSWORD=admin
OPENSEARCH_DASHBOARDS_URL=http://localhost/analytics

# Frontend (.env)
VITE_OPENSEARCH_DASHBOARDS_URL=http://localhost/analytics
```

### Setting Up the Index Template

After starting OpenSearch, create the security findings index template:

```bash theme={null}
cd backend
bun run setup:opensearch
```

This creates the `security-findings-*` index template with proper mappings for workflow execution data.

### Using Analytics

1. **Analytics Sink Component**: Add the "Analytics Sink" component to your workflows to index output data
2. **Dashboards Link**: Access OpenSearch Dashboards from the Studio sidebar
3. **Query API**: Use the `/api/analytics/query` endpoint to query indexed data programmatically

### Analytics Service Endpoints

| Service               | URL                                                        | Notes                        |
| --------------------- | ---------------------------------------------------------- | ---------------------------- |
| OpenSearch            | [http://localhost:9200](http://localhost:9200)             | Search engine API            |
| OpenSearch Dashboards | [http://localhost/analytics/](http://localhost/analytics/) | Visualization UI (via nginx) |

<Note>
  The analytics stack is optional. If OpenSearch is not configured, the Analytics Sink component will gracefully skip indexing and log a warning.
</Note>

***

## Production Deployment

For production, use the Docker-based deployment:

```bash theme={null}
# Start production environment with cached images
just prod

# Or build and start
just prod build
```

Production endpoints (all via nginx on port 80):

* Frontend: [http://localhost/](http://localhost/)
* Backend API: [http://localhost/api/](http://localhost/api/)
* Analytics: [http://localhost/analytics/](http://localhost/analytics/)
* Temporal UI: [http://localhost:8081](http://localhost:8081)

***

## Common Commands

### Development

```bash theme={null}
just dev          # Start development environment
just dev stop     # Stop everything
just dev logs     # View application logs
just dev status   # Check service status
just dev clean    # Stop and remove all data
```

### Production

```bash theme={null}
just prod          # Start with cached images
just prod build    # Rebuild and start
just prod stop     # Stop production
just prod logs     # View production logs
just prod status   # Check production status
just prod clean    # Remove all data
```

### Infrastructure

```bash theme={null}
just infra up     # Start infrastructure only
just infra down   # Stop infrastructure
just infra logs   # View infrastructure logs
just infra clean  # Remove infrastructure data
```

### Utilities

```bash theme={null}
just status       # Show status of all services
just db-reset     # Reset database (drops all data)
just build        # Build images only
```

***

## Troubleshooting

### macOS: Native module build errors

```bash theme={null}
export SDKROOT=$(xcrun --show-sdk-path)
bun install
```

### Services not starting

```bash theme={null}
# Check status
just dev status

# Restart everything
just dev stop
just dev
```

### Database connection failed

```bash theme={null}
# Run migrations manually
bun run migrate

# Check database logs
just infra logs
```

### PM2 processes failing

```bash theme={null}
# Check environment files
ls -la */.env

# Restart everything
just dev stop
just dev

# Check logs
just dev logs
```

### Frontend not loading

```bash theme={null}
# Check if dev server is running
just dev status

# Start manually if needed
bun --cwd frontend dev
```

### Temporal namespace missing

```bash theme={null}
docker compose -p shipsec --profile setup up -d
```

### Clean rebuild

```bash theme={null}
just dev stop
just infra clean
just infra up
bun run migrate
just dev
```

***

## Manual Setup (Optional)

If you need more control over the setup process:

### Infrastructure only

```bash theme={null}
just infra up
just status
just infra logs
```

### Applications only

```bash theme={null}
pm2 start pm2.config.cjs
pm2 status
pm2 logs
```

***

## Getting Help

* Check [GitHub Issues](https://github.com/shipsecai/studio/issues) for similar problems
* Review logs with `just dev logs` for error details
* Email support at [support@shipsec.ai](mailto:support@shipsec.ai)


# Quickstart
Source: https://docs.shipsec.ai/quickstart

Get ShipSec Studio running in 5 minutes

## Prerequisites

* Docker Desktop (≥8 GB RAM)
* [Bun](https://bun.sh) runtime
* [just](https://github.com/casey/just) command runner

## Quick Setup

<Steps>
  <Step title="Clone and initialize">
    ```bash theme={null}
    git clone https://github.com/shipsecai/studio.git
    cd studio
    just init
    ```

    This automatically installs dependencies and creates environment files.
  </Step>

  <Step title="Start development environment">
    ```bash theme={null}
    just dev
    ```

    This starts all infrastructure (Postgres, Temporal, MinIO, Loki), runs migrations, and launches the full stack with hot-reload.
  </Step>

  <Step title="Open ShipSec Studio">
    Navigate to [http://localhost](http://localhost) and sign in or create an account.
  </Step>
</Steps>

## Access Points

All services are accessible through nginx on port 80:

| Service     | URL                                            |
| ----------- | ---------------------------------------------- |
| Frontend    | [http://localhost/](http://localhost/)         |
| Backend API | [http://localhost/api/](http://localhost/api/) |
| Temporal UI | [http://localhost:8081](http://localhost:8081) |

<Note>
  Individual service ports are available for debugging (frontend: 5173, backend: 3211) but should not be used in normal development.
</Note>

## Your First Workflow

<Steps>
  <Step title="Create a new workflow">
    Click **New Workflow** in the dashboard. Name it "Subdomain Discovery".
  </Step>

  <Step title="Add components">
    * Drag **Manual Trigger** to the canvas
    * Drag **Subfinder** component to the canvas
    * Connect the trigger output to the Subfinder input
  </Step>

  <Step title="Run the workflow">
    Click **Save**, then **Run**. Watch the execution unfold in real-time with streaming logs and terminal output.
  </Step>
</Steps>

## Common Commands

```bash theme={null}
# View logs
just dev logs

# Check status
just dev status

# Stop everything
just dev stop

# Restart
just dev stop && just dev
```

## Next Steps

<CardGroup>
  <Card title="Full Installation" icon="download" href="/installation">
    Detailed setup and configuration options
  </Card>

  <Card title="Command Reference" icon="terminal" href="/command-reference">
    All available just commands
  </Card>

  <Card title="Components" icon="cube" href="/components/overview">
    Explore available components
  </Card>

  <Card title="Architecture" icon="sitemap" href="/architecture">
    Understand the system design
  </Card>
</CardGroup>