No description
560ce8d798* feat(web-app): add ingestion health dashboard with prometheus metrics * feat(docs): add apps documentation and integrate into layout - Introduced a new documentation section for apps, including an index page and individual pages for `api-rs` and `mcp-rs`. - Updated the layout to incorporate the apps documentation into the existing page tree structure. - Enhanced the page resolution logic to support navigation between docs and apps, ensuring a seamless user experience. * fix(tasks): mark ingestion health dashboard as completed and update route definitions - Updated tasks.md to reflect the completion of the authenticated ingestion health dashboard. - Adjusted routeTree.gen.ts to ensure the '/dashboard/protect-examples' route is correctly defined in multiple sections, maintaining consistency across the routing structure. * fix(biome): update ignore patterns to exclude routeTree.gen.ts from version control * feat(tests): enhance load performance tests with concurrency limits - Updated load performance tests in `load_performance.rs` and `performance.rs` to implement concurrency limits using `OperationTracker`. - Introduced constants for concurrency limits in both REST and MCP tests to ensure expected peak values are enforced. - Enhanced logging and assertions to validate concurrency behavior under load, improving test reliability and accuracy. - Added new testing phase for session validation in the Rust migration documentation. * feat(observability): surface exporter health and incidents on dashboard * feat(docs): update Rust migration documentation with design decisions and task completion - Added design decisions regarding incident severity calculation, exporter health status embedding, and Sentry transaction label usage. - Marked the task for linking dashboard incidents to Sentry traces and showing exporter health/error states as completed in tasks.md. * Reset job log last event timestamp on stream restart * feat: drop python deps * feat: enhance AGENTS.md with frontend design system guidelines - Added a comprehensive design system for frontend and terminal UI work, emphasizing dense information architecture and minimal visual language. - Included specific guidelines on typography, color coding, whitespace usage, and data density to align with the overall design philosophy. * feat(auth): consolidate authentication across services with Better Auth - Updated the current specification to reflect the new focus on authentication consolidation. - Added a requirements document detailing the feature overview, user stories, acceptance criteria, and functional/non-functional requirements for the Better Auth integration. - Created a README for the new auth consolidation specification to track progress and status. * feat: extend prompts to better accept args * chore(spec): record auth consolidation approvals * docs: record auth audit completion * feat(mcp-rs): enforce Better Auth scopes for MCP HTTP routes (mcp:invoke, mcp:watch)\n\n- Extract identity via axum Extension and forbid when missing scopes\n- Add core-rs AuthIdentity::has_permission to read API key perms with JWT fallback\n- Update spec checklist and implementation log for MCP auth enforcement * feat(api-rs): enforce Better Auth permissions for /metrics endpoints (metrics:read); replace legacy bearer checks and update tests - Require AuthIdentity via middleware and check metrics:read - Keep metrics.disable_auth and enabled gate for dev/ops - Mark spec Phase 2 task complete and log session * chore(api-rs): resolve clippy warnings by removing unused imports; tests remain green * fix(api-rs): honor metrics.disable_auth by skipping AuthLayer on /metrics routes and accepting optional identity - Split router into protected_router and metrics_router - Apply AuthLayer only to protected routes; conditionally to metrics - Make metrics handlers accept Option<Extension<Arc<AuthIdentity>>> and validate permissions when required - Adjust tests to new signatures; all pass * docs(spec): log fix for metrics.disable_auth bypass and router/layer changes * feat(core-rs,api-rs): deprecate metrics bearer token config with migration shim and warnings; switch enforcement to Better Auth\n\n- Drop strict requirement for metrics.bearer_tokens_sha256 when metrics enabled\n- Add deprecation warnings in config validation and api startup logs\n- Update spec tasks and implementation log * docs(spec): mark implemented tasks complete in 003-auth-consolidation (customAPIKeyValidator, shared auth crate, shared middleware) * fix(core-rs): forbid metrics.disable_auth in production; add validation tests and update 003 spec task status * feat: propagate AuthIdentity into metrics/log exporters, add bounded org_id labels * feat(core-rs,telemetry): add auth success/failure metrics and structured logs; update spec tasks and implementation log - AuthLayer emits auth_success_total/auth_failure_total with bounded labels and structured logs\n- Telemetry registers metric descriptions for Prometheus exposition\n- Spec 003 tasks updated; implementation log appended * docs(spec): align 003 with API-key-only direction; mark dual-mode SHA-256 migration as [legacy]; add purge tasks and web-app env cleanup - requirements: direction update; dual-mode marked [legacy]\n- design: remove remaining sha-256 mentions; clarify deletion of deprecated fields\n- tasks: add purge tasks; mark dual-mode + dependency as [legacy] * feat(auth): dual‑mode metrics auth (Better Auth + legacy SHA‑256) with deprecation logs - Add legacy metrics SHA‑256 fallback to AuthLayer (metrics endpoints only) with constant‑time digest checks and WARN logs per use. - Wire api-rs metrics router to legacy-aware layer; maintain Better Auth as primary. - Emit startup deprecation warning when legacy digests configured. - Update spec tasks with Better Auth verification sources. - Log stabilization steps in current spec implementation log. Security/Scope: - Legacy fallback is strictly limited to /metrics and /metrics/snapshot. - No changes to protected routes; all other endpoints require Better Auth credentials. Notes: - Left rustfmt/clippy suggestions un-applied to avoid mixing formatting into this change. * fix(mcp-rs,core-rs): fix pre-push failures (test arg, duplicate attribute) and apply rustfmt - mcp-rs tests: pass Arc<AuthClient> to HttpTransport::new and set FIRECRAWL_API_KEY for AppConfig::load. - core-rs config tests: remove duplicate #[test] attribute. - apply cargo fmt across touched files to satisfy fmt hook. * test(mcp-rs): update tests for HttpTransport::new signature; provide AuthClient via AppConfig::load * fix(core-rs): satisfy clippy (needless_question_mark) in AuthLayer legacy path * chore(api-rs): remove unused HeaderMap import in tests to satisfy clippy * fix(mcp-rs/tests): prevent double-counted ingest concurrency in performance test by removing inner OperationGuard\n\nchore(moon): mark api-rs:serve and mcp-rs:serve as local and non-cacheable to avoid hashing/remote runs for long-lived processes\n\nfix(api-rs): drop unused "hex" dependency from Cargo.toml\n\nchore(web-app/metrics): document trusted upstream payload for MetricsSnapshot; add comment to replace with runtime validation if needed\n\nfix(web-app/auth): always reconcile admin API key on ensure; reuse cached hash when key unchanged to avoid unnecessary hashing * refactor(core-rs): extract constant_time_eq into shared crypto module and reuse in config and auth * style(core-rs): rustfmt after extracting crypto module * feat(mcp-rs): protect /metrics with AuthLayer dual-mode (Better Auth + legacy SHA-256 fallback) and remove manual header verifier * docs: document prometheus admin key usage * docs: stabilize observability docs pipeline * feat(core-rs): integrate `subtle` crate for constant-time comparisons and enhance production environment guard for metrics authentication * feat(core-rs, mcp-rs): add unit tests for shared auth crate and integrate wiremock for HTTP mocking; enhance auth client verification with BetterAuthMock * fix: enforce metrics auth and stabilize mcp tests * chore(moon): shim bun path for bun-driven tasks * chore(mcp-rs): fmt * test(core-auth): provide config helper for clippy compliance * feat(web-app): update font styles to use Geist Mono and add preconnect links for font optimization * feat(web-app): enhance card component with variant support and update styles for improved readability * feat(web-app): enhance ingestion dashboard and job log features with improved state management and accessibility options * feat(moon): add init task for project setup and update package.json for initialization command * feat(core-auth): implement dual-mode migration for legacy SHA-256 tokens with deprecation logging and metrics tracking * feat(husky): integrate husky for Git hooks and remove husky-runner; update package dependencies * feat(husky): enhance pre-push hook with detailed logging and common fix suggestions * refactor(core-auth): streamline header insertion and tidy up test imports for improved readability * chore(husky): remove pre-commit hook and enhance pre-push logging with debug output and helpful error messages * feat(moon): migrate all moon configs from YAML to PKL format for type-safety and programmability * fix(moon): update project name from "NIA Web App" to "better-slop web app" for clarity * feat(biome): add useSortedAttributes option and update various linting rules for improved code quality * feat(moon): replace husky with moonrepo's built-in VCS hooks for pre-push validation * style(ingestion-dashboard): adjust spacing in JobLogPanel for improved layout consistency * feat(moon): add Rust version 1.89.0 and db-studio task for Drizzle Studio execution * feat(spec): update current spec to 004-remove-bloat and add README for new feature * ci: install pkl during workflows * fix: guard latency tooltip formatter --------- Co-authored-by: cau1k <zero@composer.group> Co-authored-by: codex <codex@example.com> |
||
|---|---|---|
| .github/workflows | ||
| .llms | ||
| .moon | ||
| .vscode | ||
| apps | ||
| docs | ||
| ops/prometheus | ||
| packages | ||
| .gitignore | ||
| .prototools | ||
| AGENTS.md | ||
| biome.jsonc | ||
| bun.lock | ||
| Cargo.lock | ||
| Cargo.toml | ||
| docker-compose.yml | ||
| Dockerfile | ||
| moon.pkl | ||
| package.json | ||
| README.md | ||
| rust-analyzer.toml | ||
Better Slop MCP
Firecrawl-powered documentation and repo ingestion stack composed of:
- Axum REST API (
apps/api-rs) – Owns Firecrawl orchestration, queueing, storage, and vector search. All heavy work lives here. - Rust MCP server (
apps/mcp-rs) – Exposes the REST API through a Model Context Protocol tool plus a resource reader, so LLM clients never touch the underlying API directly. - TanStack Start dashboard (
apps/web-app) – React-based control surface for developers to trigger crawls, inspect indexed data, and demo AI tooling.
Shared logic (configuration, queue manager, storage, embeddings) lives in packages/core-rs and is
consumed across the Rust services.
Tooling
- moon orchestrates tasks (
moon run …) across the monorepo. The workspace toolchain in.moon/toolchain.ymlpins Bun and tells moon which proto release to bootstrap automatically when tasks run (docs). - proto remains available if you prefer its shims or
activation workflows;
.prototoolsmirrors the same versions, but prefixing commands withprotois no longer required for day-to-day work. - Bun drives the Vite development server and builds.
- TanStack Start + Tailwind CSS v4 drive
the new dashboard running out of
apps/web-app. - Biome runs from the monorepo root via
biome.jsonc. It covers all TypeScript/JavaScript projects with shared rules. The organization card component is temporarily excluded with a root override because Biome 2.2.x has a known parser bug there—we'll drop the override once upstream ships a fix.
Project Layout
apps/
api-rs/ # Axum REST API
mcp-rs/ # MCP adapter that proxies to the API
web-app/ # TanStack Start dashboard (Vite + Bun)
docs/ # Next.js documentation site (Fumadocs)
packages/
core-rs/ # Shared Rust crate (config, queue, storage, embeddings, schemas)
types-rs/ # Shared Rust types
storage/ # Default local storage + vector index (gitignored)
Getting Started
1. Toolchain
Install the moon CLI and ensure moon is on your PATH. On
macOS/Linux you can use the official installer:
bash <(curl -fsSL https://moonrepo.dev/install/moon.sh)
Once moon is installed, running any project task will download the pinned proto and Bun toolchains
defined in .moon/toolchain.yml automatically
(docs).
If you still prefer proto-specific workflows, .prototools mirrors the same versions so proto install continues to work, but it's optional.
2. Environment
Copy the example env files and fill in secrets:
cp apps/web-app/.env.example apps/web-app/.env
cp docs/.env.example docs/.env
The Rust services load configuration via dotenvy, so add overrides to a .env file at the
repository root (or export them in your shell) to adjust ports, storage, or auth endpoints.
3. Bootstrap dependencies
moon run root:setup
This installs Bun dependencies for the dashboard and docs site.
4. Optional services
Start the local S3-compatible stack (MinIO) as needed:
moon run root:docker-up
Tear it down with moon run root:docker-down when finished.
5. Database tooling (optional)
The dashboard's Drizzle migrations are exposed through a single Moon task with passthrough
arguments. Run the commands from the repo root once you have an .env in apps/web-app/:
# Generate SQL migrations from schema changes
moon run web-app:db -- generate
# or: bun run db -- generate
# Push schema updates to your Neon/Postgres instance
moon run web-app:db -- push
# or: bun run db -- push
# Launch Drizzle Studio locally
moon run web-app:db -- studio
# or: bun run db -- studio
All invocations load .env automatically via bun --env-file=.env drizzle-kit.
6. Run the stack locally
moon run root:dev
Moon parallelizes the Rust API (api-rs:serve, defaults to :9000), the Rust MCP server
(mcp-rs:serve, defaults to :8001), and the Vite dashboard (web-app:dev on :5050). Start MinIO
separately via moon run root:docker-up if you
need local object storage. Run the docs site separately with moon run docs:dev.
7. Lint & type-check
moon run root:lint
moon run root:typecheck
These aggregate cargo checks/clippy across the Rust crates and run Biome for every frontend package
(apps/web-app, docs, shared components). If you need to lint a single app directly, the same
root config powers moon run web-app:lint or moon run docs:lint.
8. Cleanup
moon run root:clean
This runs cargo clean, drops cached assets (including apps/web-app/.output), and tears down the
docker compose stack. Run it before switching branches or when cached state gets out of sync.