# 2026-09-10 — WHOLE-DAY RECORD ## 晴天记录舱 · Project Qingtian / FRO ## FRO Watchdog & IEI Server Event Logging Development Log --- **Date:** 10 September 2026 **Record type:** Whole-day project development record **Main project:** Project Qingtian / 晴天 **Main module:** FRO → Watchdog **Project directory:** `~/RemoteOffice` **Server application:** `remoteoffice` / FastAPI / Uvicorn **Application port:** `8000` --- # 1. TODAY'S MAIN OBJECTIVE Today's main objective was to establish an automatic watchdog for the Qingtian FRO service running on the IEI server. The intended protection is: ```text FRO healthy ↓ Continue normal operation FRO becomes unresponsive ↓ Watchdog detects failed health checks ↓ 3 consecutive failures ↓ Restart remoteoffice container ↓ Check FRO recovery ``` A second objective was to establish an event-log foundation for later integration with the Dashboard's **IEI Server Status** panel. --- # 2. FRO WATCHDOG IMPLEMENTATION The watchdog was implemented as a systemd timer/service pair. ## Watchdog script ```text /usr/local/bin/fro-watchdog.sh ``` The script checks: ```text http://127.0.0.1:8000/dashboard ``` The protected Docker container is: ```text remoteoffice ``` Failure state is stored in: ```text /run/fro-watchdog.failures ``` --- # 3. SYSTEMD WATCHDOG The timer was configured as: ```text fro-watchdog.timer ``` The timer runs approximately every 30 seconds. The service is: ```text fro-watchdog.service ``` The timer was enabled and started with systemd. The timer was verified using: ```bash systemctl list-timers --all | grep fro-watchdog ``` Observed executions included: ```text 16:25:51 16:26:21 16:26:52 ``` This confirms the watchdog is being executed repeatedly at the intended interval. --- # 4. HEALTH CHECK LOGIC The watchdog performs an HTTP health check against: ```text http://127.0.0.1:8000/dashboard ``` When the check succeeds: ```text FAILURES = 0 ``` When the check fails: ```text FAILURES = FAILURES + 1 ``` The configured recovery threshold is: ```text 3 consecutive failures ``` This avoids restarting FRO because of a single temporary connection failure. --- # 5. AUTOMATIC RESTART LOGIC After 3 consecutive failed checks: ```text FRO UNRESPONSIVE ↓ docker restart remoteoffice ↓ wait 5 seconds ↓ check FRO again ``` If FRO recovers: ```text FRO RECOVERED ``` If FRO remains unavailable: ```text FRO STILL DOWN ``` The existing Docker configuration also contains: ```text restart: unless-stopped ``` That provides a separate container-exit recovery mechanism. --- # 6. WATCHDOG VERIFICATION The watchdog was manually executed: ```bash sudo /usr/local/bin/fro-watchdog.sh ``` Result: ```text /run/fro-watchdog.failures 0 ``` This confirms the current FRO service is healthy and the watchdog did not trigger an unnecessary restart. The systemd service was also observed executing successfully: ```text Starting Qingtian FRO Watchdog... fro-watchdog.service: Deactivated successfully. Finished Qingtian FRO Watchdog. ``` Multiple executions completed without failure. --- # 7. EVENT LOG WRITER An event-log writer was created: ```text /usr/local/bin/fro-event-log.sh ``` Event log destination: ```text /var/log/qingtian-fro-events.log ``` The writer records: ```text YYYY-MM-DD HH:MM:SS | EVENT | DETAIL ``` Example: ```text 2026-09-10 16:30:38 | SERVER ONLINE | IEI server is running ``` --- # 8. EVENT LOG TEST A test event was written: ```bash sudo /usr/local/bin/fro-event-log.sh "FRO TEST" "Event logging test" ``` The log then contained: ```text 2026-09-10 16:30:38 | SERVER ONLINE | IEI server is running 2026-09-10 16:31:45 | FRO TEST | Event logging test ``` This confirms the event writer is operational. --- # 9. WATCHDOG → EVENT LOG INTEGRATION The Watchdog was updated to call the Event Log writer. The integration records meaningful lifecycle events instead of writing an event every 30 seconds during normal operation. Supported events are: ```text FRO UNRESPONSIVE FRO RESTARTED FRO RECOVERED FRO STILL DOWN ``` The intended event sequence during an actual FRO failure is: ```text 3 failed health checks ↓ FRO UNRESPONSIVE ↓ remoteoffice restarted ↓ FRO RESTARTED ↓ FRO becomes available ↓ FRO RECOVERED ``` If recovery fails: ```text FRO STILL DOWN ``` --- # 10. IMPORTANT DESIGN DECISION The Dashboard should not receive a new event every 30 seconds while FRO is healthy. Therefore: ```text Healthy check → no event State change / recovery / restart → event ``` This keeps the future Dashboard event history meaningful. --- # 11. WATCHDOG BACKUP Before modifying the watchdog, a backup was created: ```text /usr/local/bin/fro-watchdog.sh.bak_20260910_before_eventlog ``` The backup was verified. Original and backup were both present and had the same file size: ```text 1166 bytes ``` This confirms the backup was created before the event-log modification. --- # 12. SCRIPT SYNTAX VERIFICATION The updated script was checked using: ```bash sudo bash -n /usr/local/bin/fro-watchdog.sh ``` Result: ```text syntax=0 ``` Therefore the Bash syntax is valid. --- # 13. NORMAL OPERATION TEST AFTER MODIFICATION The updated watchdog was executed manually: ```bash sudo /usr/local/bin/fro-watchdog.sh ``` No error was produced. The failure counter remained: ```text 0 ``` The event log did not receive a false recovery/restart event during the healthy check. This confirms normal operation remained unchanged after the event-log integration. --- # 14. FINAL CURRENT ARCHITECTURE ```text IEI SERVER │ ▼ systemd fro-watchdog.timer │ every ~30 sec │ ▼ Check FRO /dashboard │ ┌────────┴────────┐ │ │ HEALTHY FAILED │ │ failures = 0 failures + 1 │ 3 consecutive failures │ ▼ restart remoteoffice │ ▼ Check recovery │ │ OK FAIL │ │ RECOVERED STILL DOWN │ ▼ Event Log records ``` --- # 15. CONSULTATION / ENGINEERING ASSESSMENT ## 15.1 Module Closure The **FRO Watchdog Module is complete and should be closed**. The current implementation provides application-level protection for FRO without adding unnecessary complexity. No artificial failure test is recommended at this stage because intentionally stopping the production FRO container would interrupt the working system. --- ## 15.2 Protection Boundary The current watchdog protects: ```text FRO application remoteoffice Docker container ``` It does not protect against: ```text Complete IEI OS freeze Kernel crash Physical machine lock-up Power failure ``` A future **IEI OS / Hardware Watchdog** should be treated as a separate module. --- ## 15.3 Dashboard IEI Server Status The Dashboard already has an IEI Server Status visual area, but it should later be connected to the real event log. Recommended future architecture: ```text /var/log/qingtian-fro-events.log ↓ FRO backend API ↓ Dashboard IEI Server Status ``` The Dashboard should eventually display real events such as: ```text Server Online Server Offline Server Reconnected FRO Restarted System Upgrade ``` The existing placeholder event rows should be replaced only when the real event source is ready. --- ## 15.4 System Upgrade Events System upgrade events should be implemented separately from the FRO watchdog. Reason: ```text Watchdog = FRO availability / recovery System Event Log = IEI server lifecycle ``` Keeping these responsibilities separate will make the system easier to maintain. --- # 16. IMPORTANT PATHS ### Watchdog ```text /usr/local/bin/fro-watchdog.sh ``` ### Watchdog backup ```text /usr/local/bin/fro-watchdog.sh.bak_20260910_before_eventlog ``` ### Event writer ```text /usr/local/bin/fro-event-log.sh ``` ### Event log ```text /var/log/qingtian-fro-events.log ``` ### systemd timer ```text /etc/systemd/system/fro-watchdog.timer ``` ### systemd service ```text /etc/systemd/system/fro-watchdog.service ``` --- # 17. CURRENT STATUS | Item | Status | |---|---| | FRO watchdog script | Complete | | systemd watchdog timer | Complete | | 30-second health checking | Complete | | 3-failure restart threshold | Complete | | Docker container restart | Complete | | Recovery check | Complete | | Event log writer | Complete | | Watchdog event integration | Complete | | Watchdog backup | Complete | | Bash syntax check | PASS | | Normal health test | PASS | | Dashboard real event integration | Next module | | System upgrade event recording | Future refinement | | IEI OS / hardware watchdog | Future module | --- # 18. NEXT SESSION Recommended order: ```text 1. Connect Dashboard IEI Server Status to real Event Log 2. Replace placeholder IEI events 3. Add actual Server Online / Offline / Reconnected events 4. Add System Upgrade event recording 5. Later evaluate IEI OS / hardware watchdog ``` --- # 19. END-OF-DAY RESULT Today's FRO Watchdog work successfully established: ```text Automatic monitoring ↓ Failure detection ↓ Automatic FRO container restart ↓ Recovery verification ↓ Lifecycle event logging ``` The module is now: **FRO Watchdog — COMPLETE / CLOSED** The next stage is not more watchdog logic. The next stage is: **IEI Server Status real-event integration.** --- # END OF WHOLE-DAY RECORD **10 September 2026** **晴天记录舱 · Project Qingtian / FRO** **FRO Watchdog Development Day**