Build Status Iteration - 2026-03-15
Completed Modules
- Core orchestration scaffolding in core/orchestrator
- Service registry with startup dependency order
- Internal service orchestration API routes in developer API
- cPanel-style hsctl service subcommands
- WSL environment test loop script
Modules In Progress
- DNS cluster implementation logic
- Container runner implementation logic
- Recovery manager active restart loop implementation
- Service-specific integration tests (mail, dns, ftp functional flows)
Modules Missing
- Full panel/frontend and panel/backend canonical split under panel/
- Dedicated service folders for every stack component with runtime/config/data/log segregation
- End-to-end mail and dns functional automation
- Production secrets bootstrap automation in systemd environment files
Errors Detected
- None in source validation for newly added files
Fixes Applied
- Added dependency-aware orchestrator module and internal API
- Added hsctl service command group for start/stop/restart/status
- Added scripts/test_environment.sh for repeated WSL validation
Next Steps
- Run test_environment.sh in WSL and collect failure points.
- Add orchestrator-backed restart policy engine under core/recovery-manager.
- Implement dns-cluster sync and health checks with config-driven master/slave roles.
- Add production env file wiring and lock down default secrets before launch.
Iteration Update (same day)
Completed Modules
- Added executable recovery manager in
core/recovery-manager/recovery_manager.py. - Added DNS cluster status/sync script in
core/dns-cluster/sync.sh. - Added
hsctl dns status|syncandhsctl recovery run-once|statuscommands. - Expanded WSL test script with orchestrator/recovery/DNS script checks.
Validation Results
- WSL environment test:
HEALTHY (12 checks passed). - Internal orchestrator API validated over localhost proxy.
hsctl dns statusandhsctl recovery run-oncevalidated from WSL.
Current Runtime Risks
- Remaining launch blockers for this iteration: none in WSL runtime test loop.
Runtime Fixes Applied
- Installed and enabled
memcached. - Installed
dnsutils(providesdig). - Added
core/dns-cluster/configure_wsl.shand applied WSL-safe PowerDNS config:- DNS listener moved to
127.0.0.1:5300 - PowerDNS web/API enabled on
127.0.0.1:8053
- DNS listener moved to
- Fixed line-ending portability in deployed scripts (
sed -i 's/\r$//').
Latest Validation
- WSL environment test:
HEALTHY (14 checks passed). - PowerDNS API listener confirmed on
127.0.0.1:8053. - Recovery manager run-once and hsctl dns/recovery command groups execute successfully.
Distributed License Runtime (Implemented)
- Added executable client:
core/license-client/license_client.py. - Added system API endpoints:
POST /api/system/license/cache-keyGET /api/system/license/runtime-status
- Added config options in developer API:
HSDEV_LICENSE_VALIDATE_PATHHSDEV_LICENSE_CACHE_PATHHSDEV_LICENSE_GRACE_HOURS
Distributed License Validation Result
- Cache key write endpoint works and persists to
/usr/local/hspanel/configs/license.cache. - Runtime status endpoint returns structured failover state when central API is unreachable (
status=unreachablewithout grace window).
Recovery Scheduling (Implemented)
- Added systemd unit:
systemd/hostingsignal-recovery.service. - Added systemd timer:
systemd/hostingsignal-recovery.timer. - Timer enabled in WSL runtime (
hostingsignal-recovery.timeractive), executing one-shot recovery cycles every 30 seconds.
Dashboard Runtime Visibility (Implemented)
- Dashboard now displays distributed license runtime source/status/grace details from
/api/system/license/runtime-status.
Latest Validation Snapshot
- Localhost login endpoint healthy: HTTP 200.
- Internal preflight and license runtime endpoints healthy through
/devapiproxy. - WSL environment test script now passes 16 checks (
HEALTHY).
Launch Hardening Automation (Implemented)
- Added env support for distributed licensing values in API config:
HSDEV_LICENSE_SERVER_URLHSDEV_LICENSE_API_KEY
- Added
scripts/generate_production_env.shto generate hardened production env file with:- random JWT secret
- random bootstrap admin password
- random WHMCS shared/HMAC secrets
- production mode enabled
- PostgreSQL DSN placeholder (non-SQLite baseline)
- Added systemd unit template
systemd/hostingsignal-devapi.servicewith optionalEnvironmentFilesupport. - Added
scripts/apply_devapi_production_env_wsl.shto:- copy env file to
/etc/hostingsignal/hostingsignal-devapi.env - create drop-in override under
/etc/systemd/system/hostingsignal-devapi.service.d/override.conf - daemon-reload + restart + health check the live dev API service
- copy env file to
- Updated README with launch hardening checklist using generated env + preflight verification flow.
Container Runner (Implemented)
- Added executable runtime module:
core/container-runner/container_runner.py. - Added secured container API router:
developer-panel/api/containers.py. - Mounted container routes in
developer-panel/api/main.pyunder/api/containers/*. - Added
hsctl containercommand group:statuslistrunstartstopremovelogs
- Expanded WSL environment checks to assert container runner script presence.
- Added web dashboard
Containersview with runtime availability and container inventory table.
DNS Replication Verification (Implemented)
- Upgraded
core/dns-cluster/sync.shwithverifymode. verifyperforms SOA serial comparisons between master and configured slave nodes.- Returns non-zero on drift/no-response to support CI/automation gating.
- Added
hsctl dns verify --zone <domain>command.
Preflight Coverage Extension (Implemented)
- Launch preflight now includes
container_runtime_accesswarning check. - Detects missing runtime or inaccessible Docker/Podman daemon before launch.
- Container runner now returns actionable hint when Docker socket permission is denied.
WSL Iteration Deployment Automation (Implemented)
- Added
scripts/deploy_iteration_wsl.shfor one-command rollout of this iteration to/usr/local/hspanel. - Script copies updated API/core/web/CLI files, rebuilds web, restarts services, and verifies:
/api/health- login +
/api/system/preflight /api/containers/status- full
scripts/test_environment.shloop
Container Runtime Permission Automation (Implemented)
- Added
scripts/fix_container_runtime_permissions_wsl.sh. - Script ensures
dockergroup exists, adds target user, restarts docker service, and verifiesdocker infoaccess.
Test Reliability Hardening (Implemented)
- Updated
scripts/test_environment.shto use per-runmktempfiles with cleanup trap. - Eliminates prior
/tmp/hs_test.outownership collision causing false-negative failures in non-root runs.
Test Credential Utility (Implemented)
- Added
scripts/get_test_credentials_wsl.sh. - Reads effective dev API bootstrap credentials from
/etc/hostingsignal/hostingsignal-devapi.envwith safe defaults. - Optional
--verifymode performs live login againsthttp://127.0.0.1:2087/api/auth/login.
Next Steps
- Implement distributed license runtime flow in
core/license-clientusingconfigs/license.cachewith 72-hour grace period. - Add recovery manager systemd unit/timer for continuous 30-second loops.
- Add functional DNS replication test targets and slave verification IPs for non-placeholder cluster nodes.
- Begin container-runner executable support for Docker/Podman service lifecycle.