Testing

OpenFactory runs two complementary surfaces under “Testing”: ISO image testing (every OS-image build boots in a KVM VM and runs assertions before the ISO is released) and the App Platform (deploy a Git repo or a one-line brief to a tester VM, layer on managed services, and verify with replayable scenarios or an autonomous walker).

App Platform

Deploy a Git repo to a tester VM and get a public preview URL at https://<slug>.apps.openfactory.tech, then bolt on managed Postgres, S3-compatible object storage, custom domains with TLS, encrypted secrets, bi-directional checkpoints, observability, a browser IDE, OIDC auth, template scaffolds, and autonomous walkers that discover UI bugs and emit replayable scenarios. Every capability is callable via MCP from the same coding agent that built the app.

Topic	Description
App Deployment	Deploy a Git repo as an app at `https://<slug>.apps.openfactory.tech`
App UI Testing	Record and replay GUI scenarios against any reachable URL
Prompt-to-App Agent	Turn a one-line brief plus a template into a deployed app
App Secrets and Env Vars	Encrypted per-app and account-level env management with audited reveal
Managed Databases	One-call managed Postgres with auto-injected `DATABASE_URL`
Object Storage	S3-compatible bucket per app with auto-injected `OF_S3_*` env vars
Custom Domains and TLS	Attach your own domain with verification and squatting defense
Checkpoints and Rollback	Bi-directional snapshots for safe agent-driven iteration
Observability	Requests, uptime probes, event timeline, per-VM resource usage
Web IDE	Browser-based code-server tied to your app’s VM
App Auth Service	One-call OIDC binding with auto-published auth env vars
Templates and Remix	Gallery of scaffolds plus fork-public-apps with hard-rule security
Autonomous Walker	BFS bug hunt that explores a UI and emits replayable scenarios
Walker Diffs and Tickets	Diff two walks, see what regressed, export findings as Linear/Jira/GitHub tickets
Walk-and-Fix Agent	Turn walker findings into FixIntents and materialize patches against running deployments

ISO Image Testing

Every OpenFactory image build is automatically tested in a KVM virtual machine before the ISO is made available for download.

Why Automated Testing?

Building a custom Linux image involves many components that must work together:

Kernel and bootloader configuration
Service dependencies and startup order
Network configuration
User authentication
Package compatibility

OpenFactory’s testing system verifies all these components automatically.

Test Infrastructure


┌─────────────────────────────────────────────────────────┐
│                    Test Pipeline                         │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  ┌──────────┐    ┌──────────┐    ┌──────────────────┐  │
│  │   ISO    │───▶│   KVM    │───▶│   Test Runner    │  │
│  │  Image   │    │   VM     │    │   (assertions)   │  │
│  └──────────┘    └──────────┘    └──────────────────┘  │
│                       │                    │            │
│                       │              ┌─────▼─────┐     │
│                       │              │  Results  │     │
│                       └─────────────▶│  Report   │     │
│                                      └───────────┘     │
│                                                          │
└─────────────────────────────────────────────────────────┘

Test Categories

Default Tests

Every build automatically runs:

Boot verification - System boots successfully
Login test - Users can authenticate
Package check - Requested packages installed
Network test - Basic connectivity works

Custom Assertions

Add your own verification tests:

Service running checks
Port listening verification
HTTP endpoint testing
File content validation
Command output checking

CIS Benchmarks

Security compliance testing:

CIS Debian Linux Benchmark
CIS Ubuntu Linux Benchmark
Configurable hardening levels

Test Results

After testing completes, you can view:

Pass/Fail status for each test
Detailed logs from test execution
Screenshots of the VM during testing
Timing information for each stage

Quick Links

Topic	Description
Default Tests	What’s tested automatically
Custom Assertions	Write your own tests
CIS Benchmarks	Security compliance

Test Failures

When tests fail:

Review the logs - Detailed output shows what went wrong
Check configuration - Verify your build recipe
Modify and rebuild - Make changes and try again
Ask for help - OpenFactory can suggest fixes

Failed builds provide detailed diagnostics to help identify and fix issues.