diff --git a/.gitignore b/.gitignore index f584932..7a43347 100644 --- a/.gitignore +++ b/.gitignore @@ -12,4 +12,6 @@ cache uploads # Database -*.db \ No newline at end of file +*.db + +ai_agent_master_prompt_for_building_modern_git_platform.md diff --git a/AGENTS.md b/AGENTS.md index 3c0a909..18f6163 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,6 +36,20 @@ internal/ dag.go — topological sort, ReadyJobs parser.go — .forgebucket/workflows/*.yml parser types.go — WorkflowFile, WorkflowJob, WorkflowStep structs + sbom/ — SBOM generator (fully built — Phase 4) + generator.go — CycloneDX 1.4 generation, auto on pipeline success + on-demand + cyclonedx.go — CycloneDX document model and helpers + parsers.go — Manifest parsers: go.mod, package.json, requirements.txt, Cargo.toml, Gemfile.lock, pom.xml + scanning/ — Secret scanner (fully built — Phase 4) + scanner.go — Push-triggered regex scanning, listing, dismissal + secrets.go — 15 high/medium severity secret patterns (AWS, GitHub, SSH, JWT, etc.) + vulnscan/ — Vulnerability scanner (fully built — Phase 4) + scanner.go — OSV API-backed dependency vulnerability scanning + osv.go — HTTP client for api.osv.dev/v1 + signing/ — Artifact signing (fully built — Phase 4) + keystore.go — ECDSA P-256 signing and verification, self-verifying bundles + oci/ — OCI registry (fully built — Phase 4) + registry.go — Content-addressable blob store, upload sessions, OCI Distribution Spec v1.1 gitops/ — GitOps controller (fully built — Phase 3D) controller.go — NATS subscriptions, startup, periodic ticker drift.go — CheckDrift, handlePush, periodicCheck @@ -49,7 +63,7 @@ internal/ observability/ — Prometheus metrics + health (fully built — Phase 3E) metrics.go — metric definitions, HTTP middleware, NATS watcher health.go — Check() returning HealthStatus (DB ping + NATS) - models/ — XORM structs + 14 migration files + models/ — XORM structs + 20 migration files config/ — ENV-driven config, fails fast on missing secrets events/ — NATS EventBus interface + NATSBus + NoOpBus (Healthy() bool) web/ — //go:embed target for the built React SPA @@ -84,7 +98,8 @@ Logger → RealIP → Recoverer → Metrics → CORS → CSRF → SessionAuth | 3D | GitOps controller + drift detection + auto-sync | **Complete** | | 3E | Observability (Prometheus `/metrics`, structured `/health`, repo health API) | **Complete** | | 3F | Federation handlers (ActivityPub WebFinger, actor, inbox/outbox, HTTP signatures, Follow/Accept) | **Complete** | -| 4 | AI diagnostics, signed artifacts, OCI registry, dep/secret scanning | Planned | +| 4 | SBOM generation, secret scanning, vuln scanning, signed artifacts, OCI registry, security page | **Complete** | +| 5 | AI diagnostics, deployment promotions, rollback visualization | Planned | --- @@ -111,7 +126,7 @@ This rule is non-negotiable. It prevents command injection. ### Database - XORM for all DB access. Structs in `internal/models/`. -- Migrations are numbered files in `internal/models/migrations/`. Always add a new file; never edit existing ones. Current highest: **014**. +- Migrations are numbered files in `internal/models/migrations/`. Always add a new file; never edit existing ones. Current highest: **020**. - No raw SQL strings built from user input. ### Events @@ -218,6 +233,16 @@ make lint # go vet + ESLint | `internal/domain/federation/remote.go` | FetchActor (cached remote actors), DeliverActivity | | `internal/api/handlers/secret.go` | Scoped secret management | | `internal/api/handlers/workspace.go` | Workspace + member management | +| `internal/api/handlers/secret.go` | Scoped secret management | +| `internal/api/handlers/sbom.go` | SBOM generation + download endpoints | +| `internal/api/handlers/scanning.go` | Secret leak list + dismiss endpoints | +| `internal/api/handlers/vulnscan.go` | Vulnerability scan + dismiss endpoints | +| `internal/api/handlers/oci.go` | OCI Distribution Spec v1.1 registry handler | +| `internal/domain/sbom/generator.go` | SBOM generator (CycloneDX 1.4) | +| `internal/domain/scanning/scanner.go` | Push-triggered secret scanner | +| `internal/domain/vulnscan/scanner.go` | OSV API-backed vulnerability scanner | +| `internal/domain/signing/keystore.go` | ECDSA P-256 artifact signing | +| `internal/domain/oci/registry.go` | Content-addressable OCI blob store | | `internal/api/middleware/audit.go` | Audit log middleware | | `frontend/src/ui/tokens.ts` | Design token source of truth | | `frontend/src/components/AppShell.tsx` | Root layout wrapper | @@ -232,7 +257,7 @@ make lint # go vet + ESLint ```bash cp .env.example .env # fill SESSION_SECRET and CSRF_SECRET make docker-up # PostgreSQL + NATS via Docker Compose -make migrate # run XORM migrations (currently 014) +make migrate # run XORM migrations (currently 020) make dev # Go :8080 + Vite :5173 ``` diff --git a/CHANGELOG.md b/CHANGELOG.md index e503421..8c4bf35 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,41 +9,81 @@ Versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] -### Planned — Phase 4 ( Artifacts + Git HTTP(S)/SSH Support + Releases Page) +### Planned — Phase 5 (AI Diagnostics + Deployment Promotions + Rollback Visualization) +- AI-powered pipeline failure diagnostics +- Deployment promotion workflows (manual + automated) +- Rollback visualization and timeline -### 4A. Artifacts -- Signed artifacts (Sigstore/Cosign) -- SBOM generation (CycloneDX/SPDX) -- OCI container registry -- Secret scanning (commit-level pattern detection) -- Dependency vulnerability scanning -- Cross-instance pull requests (ForgeFed ActivityPub extension) +--- -### 4B. Git HTTP(S)/SSH Support +## [1.0.0] — 2026-05-13 -### 4C. Releases Page -- Goal: - - Make releases operationally meaningful. -- Build: - - releases - - release notes - - release assets - - changelog generation - - release timelines - - release channels -- Add: - - signed artifacts - - rollback metadata - - deployment associations -- UI: - - Release page should show: - - version - - artifacts - - deployments - - timeline - - health +Phase 4 complete. SBOM generation, secret scanning, dependency vulnerability scanning, signed artifacts, and OCI registry are operational. + +### Added — SBOM Generation (`internal/domain/sbom/`) +- **`Generator`** — subscribes to `pipeline.completed` events and auto-generates CycloneDX 1.4 SBOM documents for every successful pipeline run; also supports on-demand generation via `GenerateOnDemand` +- **6 manifest parsers**: `go.mod`, `package.json`, `requirements.txt`, `Cargo.toml`, `Gemfile.lock`, `pom.xml` — lightweight line-scanning, no external parser dependencies +- **API endpoints** — `GET /sbom`, `GET /sbom/document`, `GET /runs/{runID}/sbom`, `GET /runs/{runID}/sbom/document`, `POST /sbom/generate?ref=&runID=` +- **Database** — migration `016_sbom` adds `SBOMReport` model with CycloneDX document body +- Automatic generation on pipeline completion now also fires directly from the orchestrator (not solely via NATS), ensuring SBOMs are generated even when NATS is unavailable + +### Added — Secret Scanning (`internal/domain/scanning/`) +- **`Scanner`** — subscribes to `push.received` events, scans git diffs against 15 regex patterns for high/medium severity secrets +- **Secret patterns**: AWS keys, GitHub/GitLab tokens, generic API keys, Bearer tokens, Slack tokens, Google API keys, Google service accounts, SSH private keys, JWTs, NPM tokens, PostgreSQL/Redis connection strings, generic passwords +- **API endpoints** — `GET /secrets/leaks`, `POST /secrets/leaks/{leakID}/dismiss` (repo-scoped), `GET /api/v1/secrets/leaks` (global admin) +- **Database** — migration `018_scanning` adds `SecretLeak` model + +### Added — Vulnerability Scanning (`internal/domain/vulnscan/`) +- **`Scanner`** — triggers on-demand scans against the OSV API (`api.osv.dev/v1`); supports scanning by PURL or by fetching the latest SBOM and scanning all components +- **OSV client** — HTTP client with 30-second timeout, queries OSV database for CVEs by PURL or ecosystem+name, extracts CVSS scores and fixed version ranges +- **API endpoints** — `GET /vulnerabilities`, `POST /vulnerabilities/scan`, `POST /vulnerabilities/{findingID}/dismiss` (repo-scoped), `GET /api/v1/vulnerabilities` (global admin) +- **Database** — migration `019_vulnscan` adds `VulnerabilityFinding` model +- Findings deduplicated by `(vuln_id, purl, repo_id)` + +### Added — Artifact Signing (`internal/domain/signing/`) +- **`KeyStore`** — ECDSA P-256 signing and verification; produces self-verifying `Bundle` carrying payload, signature (ASN.1 DER), and public key PEM +- `Sign(artifactID, name, rawContent)` — computes SHA-256 digest, signs, returns signed `Bundle` with key ID fingerprint +- `Verify(bundleJSON)` — extracts public key from bundle, verifies ECDSA signature, returns `VerifyResult` with key-matching check +- `Generate()` — creates ephemeral ECDSA P-256 key when `ARTIFACT_SIGNING_KEY` env var is unset (logs warning; signatures lost on restart unless persisted) +- **API endpoints** — `GET /artifacts/{artifactID}/signature`, `GET /artifacts/{artifactID}/verify` +- **Database** — migration `015_signing` adds `ArtifactSignature` model + +### Added — OCI Registry (`internal/domain/oci/`) +- **`Registry`** — content-addressable on-disk blob store implementing OCI Distribution Spec v1.1 +- Storage layout: `{root}/blobs/sha256/` for blobs, `{root}/uploads/` for in-progress uploads +- Full upload session lifecycle: start (POST), append chunk (PATCH), finalize with digest verification (PUT), cancel (DELETE), offset query (GET) +- 13 OCI distribution error codes defined (`ErrBlobUnknown`, `ErrDigestInvalid`, `ErrManifestInvalid`, etc.) +- **API handlers** (`internal/api/handlers/oci.go`, 525 lines) — full `/v2/{name}/{kind}/{ref}` routing: manifest push/get/delete, blob HEAD/get/delete, tag listing, chunked upload +- **Database** — migration `017_oci` adds `OCIRepository`, `OCIManifest`, `OCITag`, `OCIBlob`, `OCIUpload` models +- Registry is consumed by standard OCI tools (Docker, Podman, Skopeo, containerd) + +### Added — Unified Security Page (`/repos/:owner/:repo/security`) +- **`RepoSecurityPage`** — single-page view combining SBOM status, secret leak detection, and vulnerability findings +- SBOM section: displays existing SBOM metadata with download button, or "Generate SBOM" form with branch/SHA input +- Secret Leaks section: lists leaks with severity badge, pattern name, commit SHA, ref, match sample, dismiss button +- Vulnerabilities section: lists findings with CVSS severity (CRITICAL/HIGH/MEDIUM/LOW), vuln ID, score, summary, PURL, version, fix suggestion, dismiss button; "Scan now" trigger +- **Route** added in `App.tsx`, nav link added in `RepoPage.tsx` tab bar + +### Added — Pipeline Run SBOM Integration +- `PipelineRunPage` shows per-run SBOM section: metadata (components, SHA, generation time) + download button +- "Generate SBOM" button for completed/failed runs that lack one +- `useRunSBOM` / `useGenerateSBOM` hooks in `frontend/src/api/queries/sbom.ts` +- 404 from `useRunSBOM` handled gracefully (returns `null` instead of throwing) + +### Added — Database Models +- Migration `016_sbom` — `SBOMReport` (repoId, runId, sha, format, componentCount, bomDocument, generatedAt) +- Migration `017_oci` — `OCIRepository`, `OCIManifest`, `OCITag`, `OCIBlob`, `OCIUpload` +- Migration `018_scanning` — `SecretLeak` (repoId, commitSha, ref, patternName, description, severity, matchSample, dismissed, dismissedBy, dismissedAt, detectedAt) +- Migration `019_vulnscan` — `VulnerabilityFinding` (repoId, vulnId, purl, version, summary, details, cvssScore, fixedVersion, dismissed, dismissedBy, dismissedAt, detectedAt) +- Migration `020_forgefed` — repository + pull request column updates + +### Fixed +- SBOM per-run download endpoint (`/runs/{runID}/sbom/document`) was registered at the wrong router nesting level, causing a route conflict with the `GetLatestDocument` handler. Moved into the correct `/runs/{runID}` route block. +- `username` context key extraction in scanning and vulnerability handlers changed from raw string `"user"` to typed `middleware.ContextKeyUsername` +- Nil-safe `Needs` marshalling in orchestrator job creation +- Nil-safe findings response in vulnerability scan API +- `GenerateOnDemand` SBOM cache key now includes `runID` to prevent per-run generation from being shadowed by prior on-demand generation -> NOT just markdown notes. --- ## [0.9.0] — 2026-05-12 @@ -361,7 +401,8 @@ Initial development milestone. Core Git hosting, collaboration, and frontend SPA --- -[Unreleased]: https://github.com/forgeo/forgebucket/compare/v0.9.0...HEAD +[Unreleased]: https://github.com/forgeo/forgebucket/compare/v1.0.0...HEAD +[1.0.0]: https://github.com/forgeo/forgebucket/compare/v0.9.0...v1.0.0 [0.9.0]: https://github.com/forgeo/forgebucket/compare/v0.8.0...v0.9.0 [0.8.0]: https://github.com/forgeo/forgebucket/compare/v0.7.0...v0.8.0 [0.7.0]: https://github.com/forgeo/forgebucket/compare/v0.6.0...v0.7.0 diff --git a/README.md b/README.md index acbf7b0..d70e6c0 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ ForgeBucket is a self-hosted, federated developer operations platform. Where other Git platforms show you a list of files, ForgeBucket surfaces deployments, pipeline health, environment drift, and operational context directly alongside your code. Repositories are runtime systems. The dashboard is a command center. -**Status:** Active development. Phase 3F (ActivityPub federation) complete. Phase 4 (AI diagnostics + signed artifacts) is next. +**Status:** Active development. Phase 4 (signed artifacts, SBOM, secret/dependency scanning, OCI registry) complete. Phase 5 (AI diagnostics) is next. --- @@ -69,7 +69,9 @@ ForgeBucket is a self-hosted, federated developer operations platform. Where oth | Pipeline DAG visualization (frontend) | Done | | Dashboard CI command center | Done | | Pipeline log viewer (collapsible, per-step) | Done | -| Kubernetes / Firecracker runner backends | Planned (Phase 2D) | +| SBOM auto-generation on pipeline success | Done | +| Per-run SBOM download on pipeline detail page | Done | +| Kubernetes / Firecracker runner backends | Planned | | Matrix builds + reusable workflow templates | Planned | | Flaky test detection | Planned | @@ -81,9 +83,9 @@ ForgeBucket is a self-hosted, federated developer operations platform. Where oth | Unified operational timeline | Done | | Secret management (Global → Workspace → Repo → Env) | Done | | GitOps controller (drift detection + auto-sync) | Done | -| Deployment promotion workflows | Planned (Phase 4) | -| Rollback visualization | Planned (Phase 4) | -| Canary / blue-green support | Planned (Phase 4) | +| Deployment promotion workflows | Planned | +| Rollback visualization | Planned | +| Canary / blue-green support | Planned | ### Observability + Security | Feature | Status | @@ -93,10 +95,13 @@ ForgeBucket is a self-hosted, federated developer operations platform. Where oth | HTTP instrumentation middleware (latency histogram, request counter) | Done | | Per-repo operational health summary (`GET /repos/.../health`) | Done | | NATS-driven pipeline + deployment counters | Done | -| Health sparklines in repo/env pages (frontend) | Planned (Phase 4) | -| Secret scanning | Planned (Phase 4) | -| Dependency scanning | Planned (Phase 4) | -| Signed artifacts (Sigstore/Cosign) | Planned (Phase 4) | +| SBOM generation (CycloneDX 1.4, auto on pipeline success) | Done | +| Secret scanning (15 regex patterns, push-triggered) | Done | +| Dependency vulnerability scanning (OSV API backed) | Done | +| Signed artifacts (ECDSA P-256, self-verifying bundles) | Done | +| OCI Distribution Spec v1.1 registry | Done | +| Unified repo Security page (SBOM + leaks + vulns) | Done | +| Health sparklines in repo/env pages (frontend) | Planned | ### Federation | Feature | Status | @@ -110,7 +115,7 @@ ForgeBucket is a self-hosted, federated developer operations platform. Where oth | HTTP signatures (draft-cavage-http-signatures) | Done | | Follow / Accept auto-accept flow | Done | | RSA-2048 key pair lazy generation | Done | -| Cross-instance pull requests (ForgeFed) | Planned (Phase 4) | +| Cross-instance pull requests (ForgeFed) | Planned | --- @@ -153,10 +158,15 @@ ForgeBucket ├── Environment Service (environments, deployments — internal/api/handlers/environment.go) ├── Secret Manager (scoped AES-256-GCM — internal/api/handlers/secret.go) ├── Workspace Service (multi-tenant namespaces — internal/api/handlers/workspace.go) +├── SBOM Generator (CycloneDX 1.4, auto on pipeline success — internal/domain/sbom/) +├── Secret Scanner (15 push-triggered regex patterns — internal/domain/scanning/) +├── Vulnerability Scanner (OSV API-backed dependency scanning — internal/domain/vulnscan/) +├── Artifact Signing (ECDSA P-256 self-verifying bundles — internal/domain/signing/) +├── OCI Registry (Distribution Spec v1.1 blob store — internal/domain/oci/) ├── Event Bus (NATS core, NoOp fallback — internal/events/) ├── Audit Log (every mutating request — internal/api/middleware/audit.go) ├── Federation Layer (ActivityPub inbox/outbox, HTTP signatures — internal/domain/federation/) -├── Database (PostgreSQL + XORM — internal/models/) +├── Database (PostgreSQL + XORM 20 migrations — internal/models/) └── Web Frontend (React 18 + TypeScript, //go:embed — web/) ``` @@ -184,6 +194,11 @@ Logger → RealIP → Recoverer → Metrics → CORS → CSRF → SessionAuth | Code editing | CodeMirror | | Container | Docker Compose (dev) | | Federation | ActivityPub / ForgeFed (WebFinger, actor, inbox/outbox, HTTP signatures) | +| SBOM format | CycloneDX 1.4 (JSON) | +| Vulnerability data | OSV API (`api.osv.dev`) | +| Secret detection | Regex-based (15 patterns, push-triggered) | +| Artifact signing | ECDSA P-256 (ASN.1 DER, self-verifying bundles) | +| OCI storage | On-disk content-addressable blob store (Distribution Spec v1.1) | --- @@ -214,6 +229,8 @@ ForgeBucket has its own design language — intentionally distinct from GitHub a | `ARTIFACT_ROOT` | No | Artifact storage path, defaults to `../artifacts` relative to `REPO_ROOT` | | `NATS_URL` | No | NATS connection URL (e.g. `nats://localhost:4222`). If unset, event bus is no-op | | `GITOPS_RECONCILE_INTERVAL` | No | Seconds between periodic drift checks, default `300`. `0` disables the ticker | +| `OCI_ROOT` | No | Root directory for OCI Distribution Spec blob and upload storage, defaults to `../oci` relative to `REPO_ROOT` | +| `ARTIFACT_SIGNING_KEY` | No | Path to ECDSA P-256 PEM for artifact signing; auto-generates ephemeral key if unset (warns on restart) | | `INSTANCE_URL` | Yes | Public URL of this instance (no trailing slash) | | `INSTANCE_NAME` | No | Display name, default `ForgeBucket` | | `OIDC_ISSUER` | No | OIDC provider URL | @@ -250,7 +267,8 @@ ForgeBucket has its own design language — intentionally distinct from GitHub a | Phase 3D | GitOps controller + drift detection + auto-sync | Done | | Phase 3E | Observability (Prometheus `/metrics`, structured `/health`, repo health API) | Done | | Phase 3F | Federation handlers (ActivityPub WebFinger, actor, inbox/outbox, HTTP signatures) | Done | -| Phase 4 | AI diagnostics, signed artifacts, OCI registry, secret/dep scanning | Next | +| Phase 4 | Signed artifacts, SBOM, OCI registry, secret/dep scanning, security page | Done | +| Phase 5 | AI diagnostics, deployment promotions, rollback visualization | Next | ---