MVP
This commit is contained in:
Marcelo
349
DATA_PERSISTENCE_README.md
Normal file
349
DATA_PERSISTENCE_README.md
Normal file
@@ -0,0 +1,349 @@
|
||||
# Data Persistence Implementation - Complete Package
|
||||
|
||||
## Overview
|
||||
This package contains everything needed to implement **Issue #1: Data Persistence** from the optimization requirements. This enables crash recovery and session restoration for your KPI tracking system.
|
||||
|
||||
---
|
||||
|
||||
## 📦 Package Contents
|
||||
|
||||
### 📄 Documentation Files
|
||||
1. **IMPLEMENTATION_GUIDE.md** ⭐ START HERE
|
||||
- Detailed step-by-step implementation instructions
|
||||
- Flow diagrams and wiring guides
|
||||
- Testing procedures
|
||||
- Troubleshooting guide
|
||||
|
||||
2. **IMPLEMENTATION_SUMMARY.md**
|
||||
- High-level overview
|
||||
- What problems this solves
|
||||
- Expected outcomes
|
||||
- Quick reference
|
||||
|
||||
3. **NODE_CONFIGURATION.md**
|
||||
- Exact node configuration settings
|
||||
- Wiring reference
|
||||
- Complete flow diagrams
|
||||
- Common mistakes to avoid
|
||||
|
||||
4. **IMPLEMENTATION_CHECKLIST.txt**
|
||||
- Printable checklist format
|
||||
- Phase-by-phase tasks
|
||||
- Testing checklist
|
||||
- Sign-off section
|
||||
|
||||
5. **DATA_PERSISTENCE_README.md** (this file)
|
||||
- Package overview and file index
|
||||
|
||||
---
|
||||
|
||||
### 💾 Database Files
|
||||
1. **create_session_state_table.sql**
|
||||
- MySQL table schema
|
||||
- Creates `session_state` table
|
||||
- Includes indexes for performance
|
||||
|
||||
---
|
||||
|
||||
### 🔧 Modified Function Code
|
||||
1. **modified_machine_cycles.js**
|
||||
- Enhanced Machine cycles function
|
||||
- Adds database persistence
|
||||
- Throttled sync every 5 seconds
|
||||
- **Requires 3 outputs** (adds 1 new)
|
||||
|
||||
2. **modified_work_order_buttons.js**
|
||||
- Enhanced Work Order buttons function
|
||||
- Adds session management
|
||||
- Database sync on state changes
|
||||
- **Requires 5 outputs** (adds 1 new)
|
||||
|
||||
---
|
||||
|
||||
### 🆕 New Function Code (Recovery)
|
||||
1. **startup_recovery_query.js**
|
||||
- Queries for previous session on startup
|
||||
- Triggered by inject node
|
||||
|
||||
2. **process_recovery_response.js**
|
||||
- Processes database query results
|
||||
- Detects stale sessions (>24 hours)
|
||||
- Generates user prompts
|
||||
|
||||
3. **restore_session.js**
|
||||
- Restores previous session state
|
||||
- Reactivates work order
|
||||
- Restores all context variables
|
||||
|
||||
4. **start_fresh.js**
|
||||
- Starts with clean slate
|
||||
- Deactivates old session
|
||||
- Clears all counters
|
||||
|
||||
---
|
||||
|
||||
### 📚 Reference Implementation (Optional)
|
||||
1. **session_state_functions.js**
|
||||
- Modular alternative implementation
|
||||
- Can be used as reference
|
||||
- Includes all helper functions
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### For First-Time Implementation:
|
||||
1. Read **IMPLEMENTATION_SUMMARY.md** (5 min)
|
||||
2. Follow **IMPLEMENTATION_GUIDE.md** step-by-step (1-2 hours)
|
||||
3. Use **IMPLEMENTATION_CHECKLIST.txt** to track progress
|
||||
4. Reference **NODE_CONFIGURATION.md** when configuring nodes
|
||||
|
||||
### For Quick Reference:
|
||||
- **Checklist:** `IMPLEMENTATION_CHECKLIST.txt`
|
||||
- **Node Config:** `NODE_CONFIGURATION.md`
|
||||
- **SQL Queries:** `create_session_state_table.sql`
|
||||
|
||||
---
|
||||
|
||||
## 📊 What This Implements
|
||||
|
||||
### Problem Solved
|
||||
- ✅ System survives crashes and reboots
|
||||
- ✅ No data loss during power failures
|
||||
- ✅ Session restoration on startup
|
||||
- ✅ Accurate KPI tracking across restarts
|
||||
|
||||
### Technical Details
|
||||
- **Database:** MySQL/MariaDB
|
||||
- **Sync Frequency:** Every 5 seconds (throttled)
|
||||
- **Recovery Time:** <1 second on startup
|
||||
- **Data Loss Risk:** Maximum 5 seconds of data
|
||||
|
||||
### Key Features
|
||||
- Automatic session backup
|
||||
- User-prompted restoration
|
||||
- Stale session detection (>24 hours)
|
||||
- Complete audit trail
|
||||
- Zero-impact performance
|
||||
|
||||
---
|
||||
|
||||
## 📁 File Usage Matrix
|
||||
|
||||
| File | Phase | Required | Usage |
|
||||
|------|-------|----------|-------|
|
||||
| IMPLEMENTATION_GUIDE.md | All | ✅ | Primary guide |
|
||||
| IMPLEMENTATION_SUMMARY.md | Planning | ✅ | Overview |
|
||||
| NODE_CONFIGURATION.md | Implementation | ✅ | Node setup |
|
||||
| IMPLEMENTATION_CHECKLIST.txt | Implementation | ✅ | Task tracking |
|
||||
| create_session_state_table.sql | Database Setup | ✅ | Run once |
|
||||
| modified_machine_cycles.js | Implementation | ✅ | Replace code |
|
||||
| modified_work_order_buttons.js | Implementation | ✅ | Replace code |
|
||||
| startup_recovery_query.js | Implementation | ✅ | New function |
|
||||
| process_recovery_response.js | Implementation | ✅ | New function |
|
||||
| restore_session.js | Implementation | ✅ | New function |
|
||||
| start_fresh.js | Implementation | ✅ | New function |
|
||||
| session_state_functions.js | Reference | ❌ | Optional |
|
||||
| DATA_PERSISTENCE_README.md | Overview | ℹ️ | This file |
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ Time Estimates
|
||||
|
||||
| Phase | Time | Complexity |
|
||||
|-------|------|------------|
|
||||
| Database Setup | 10 min | Easy |
|
||||
| Update Machine Cycles | 15 min | Medium |
|
||||
| Update Work Order Buttons | 15 min | Medium |
|
||||
| Create Recovery Flow | 30 min | Medium |
|
||||
| UI Integration | 20 min | Medium-Hard |
|
||||
| Testing | 30 min | Easy |
|
||||
| **Total** | **2 hours** | **Medium** |
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Success Criteria
|
||||
|
||||
After implementation, you should be able to:
|
||||
- [ ] Start a work order and run cycles
|
||||
- [ ] Stop Node-RED mid-production
|
||||
- [ ] Restart Node-RED
|
||||
- [ ] See recovery prompt with accurate data
|
||||
- [ ] Restore session successfully
|
||||
- [ ] Continue production without data loss
|
||||
- [ ] See session history in database
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Testing Quick Commands
|
||||
|
||||
### Check Session State
|
||||
```sql
|
||||
SELECT * FROM session_state WHERE is_active = 1;
|
||||
```
|
||||
|
||||
### View Session History
|
||||
```sql
|
||||
SELECT session_id, cycle_count, active_work_order_id,
|
||||
FROM_UNIXTIME(updated_at/1000) as last_update
|
||||
FROM session_state
|
||||
ORDER BY updated_at DESC LIMIT 10;
|
||||
```
|
||||
|
||||
### Simulate Crash
|
||||
```bash
|
||||
sudo systemctl stop nodered
|
||||
# Wait 10 seconds
|
||||
sudo systemctl start nodered
|
||||
```
|
||||
|
||||
### Watch Logs
|
||||
```bash
|
||||
journalctl -u nodered -f
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🆘 Getting Help
|
||||
|
||||
### If Something Goes Wrong
|
||||
1. Check **IMPLEMENTATION_GUIDE.md** troubleshooting section
|
||||
2. Verify checklist completion in **IMPLEMENTATION_CHECKLIST.txt**
|
||||
3. Check Node-RED debug panel for errors
|
||||
4. Review **NODE_CONFIGURATION.md** for wiring mistakes
|
||||
5. Check database logs for SQL errors
|
||||
|
||||
### Common Issues
|
||||
- **No recovery prompt:** Check inject node timing (0.1 sec)
|
||||
- **Wrong data restored:** Check JSON escaping in SQL
|
||||
- **Sync not working:** Verify MySQL node configuration
|
||||
- **Performance issues:** Check throttle interval (5 sec)
|
||||
|
||||
---
|
||||
|
||||
## 📈 Next Steps
|
||||
|
||||
After successful implementation:
|
||||
|
||||
1. **Test thoroughly** (minimum 30 minutes)
|
||||
2. **Train operators** on recovery prompts
|
||||
3. **Monitor for 1 week** to ensure stability
|
||||
4. **Proceed to Issue #4:** Intelligent Downtime Categorization
|
||||
5. **Proceed to Issue #5:** Session Management
|
||||
|
||||
---
|
||||
|
||||
## 📋 Implementation Order
|
||||
|
||||
```
|
||||
1. Read Documentation (IMPLEMENTATION_SUMMARY.md)
|
||||
↓
|
||||
2. Create Database Table (create_session_state_table.sql)
|
||||
↓
|
||||
3. Update Machine Cycles (modified_machine_cycles.js)
|
||||
↓
|
||||
4. Update Work Order Buttons (modified_work_order_buttons.js)
|
||||
↓
|
||||
5. Create Recovery Flow (startup_recovery_query.js + others)
|
||||
↓
|
||||
6. Add UI Elements (prompts, notifications)
|
||||
↓
|
||||
7. Deploy and Test (IMPLEMENTATION_CHECKLIST.txt)
|
||||
↓
|
||||
8. Monitor and Validate (1 week)
|
||||
↓
|
||||
9. Mark Complete and Proceed to Next Issue
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Security Notes
|
||||
|
||||
- Database credentials: Use environment variables or secure storage
|
||||
- Session data: Contains work order details (not sensitive)
|
||||
- User prompts: No authentication required (internal system)
|
||||
- SQL injection: All values are escaped properly
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Development vs Production
|
||||
|
||||
### Development Environment
|
||||
- Keep debug nodes enabled
|
||||
- Test with short timeout intervals
|
||||
- Monitor logs continuously
|
||||
- Test all failure scenarios
|
||||
|
||||
### Production Environment
|
||||
- Remove or disable debug nodes
|
||||
- Use standard 5-second throttle
|
||||
- Set up automated monitoring
|
||||
- Document recovery procedures for operators
|
||||
|
||||
---
|
||||
|
||||
## 📝 Version History
|
||||
|
||||
| Version | Date | Changes |
|
||||
|---------|------|---------|
|
||||
| 1.0 | 2025-11-21 | Initial implementation package |
|
||||
|
||||
---
|
||||
|
||||
## 👥 Support Resources
|
||||
|
||||
- **Primary Documentation:** IMPLEMENTATION_GUIDE.md
|
||||
- **Optimization Requirements:** /home/mdares/.node-red/optimization_prompt.txt
|
||||
- **Node-RED Logs:** `journalctl -u nodered -f`
|
||||
- **Database Logs:** Check MySQL/MariaDB logs
|
||||
|
||||
---
|
||||
|
||||
## ✅ Pre-Implementation Checklist
|
||||
|
||||
Before starting implementation:
|
||||
- [ ] Node-RED is running and accessible
|
||||
- [ ] MySQL/MariaDB database is accessible
|
||||
- [ ] Database credentials available
|
||||
- [ ] Backup of current flows.json exists
|
||||
- [ ] Time allocated (2 hours)
|
||||
- [ ] Test environment available (or production downtime scheduled)
|
||||
- [ ] All documentation files reviewed
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Key Concepts
|
||||
|
||||
### Session ID
|
||||
Unique identifier for each production session. Created when work order starts.
|
||||
|
||||
### Throttled Sync
|
||||
Database writes limited to once every 5 seconds to reduce overhead.
|
||||
|
||||
### Forced Sync
|
||||
Immediate database write on critical events (START, STOP, complete).
|
||||
|
||||
### Stale Session
|
||||
Session older than 24 hours, automatically discarded on startup.
|
||||
|
||||
### Active Session
|
||||
Current production session with `is_active = 1` in database.
|
||||
|
||||
---
|
||||
|
||||
## 🏁 Ready to Begin?
|
||||
|
||||
1. ✅ Read this README
|
||||
2. ⬜ Read IMPLEMENTATION_SUMMARY.md
|
||||
3. ⬜ Open IMPLEMENTATION_GUIDE.md
|
||||
4. ⬜ Print IMPLEMENTATION_CHECKLIST.txt
|
||||
5. ⬜ Begin implementation
|
||||
|
||||
---
|
||||
|
||||
**Good luck with your implementation!**
|
||||
|
||||
For questions or issues, refer to the troubleshooting sections in:
|
||||
- IMPLEMENTATION_GUIDE.md (detailed)
|
||||
- NODE_CONFIGURATION.md (configuration-specific)
|
||||
- IMPLEMENTATION_CHECKLIST.txt (quick reference)
|
||||
Reference in New Issue
Block a user