# Sync

[![Commercial](https://img.shields.io/badge/License-Commercial-red.svg)](LICENSE.md)
[![Cockpit CMS](https://img.shields.io/badge/Cockpit%20CMS-Addon-blue.svg)](https://getcockpit.com)
[![Synchronization](https://img.shields.io/badge/Feature-Data%20Sync-green.svg)](#features)

> **Professional data synchronization for Cockpit CMS multi-instance deployments**

Sync enables bidirectional data synchronization between multiple Cockpit CMS instances. Perfect for staging/production workflows, content distribution, backup strategies, and multi-site content management with enterprise-grade security and reliability.

## ✨ Features

### 🔄 **Bidirectional Synchronization**
- **Push Mode**: Send content from local to remote instance
- **Pull Mode**: Receive content from remote to local instance
- **Individual Item Sync**: Sync specific content items on-demand
- **Batch Processing**: Efficient handling of large datasets with configurable batch sizes
- **Mirror Mode**: Complete replacement of target data for deployment scenarios

### 📊 **Content Types Supported**
- **Content Models**: Collection, singleton, and tree content model definitions
- **Content Data**: All content items with automatic batching (20 items per batch)
- **Assets**: Files, images, and media with automatic download/upload
- **Locales**: Multi-language configurations and settings
- **User Roles**: Permissions and access control definitions
- **Layout Components**: Layout definitions and configurations
- **Translation Projects**: Lokalize internationalization data

### 🔒 **Enterprise Security**
- **JWT Authentication**: Secure payload encryption with shared keys
- **CSRF Protection**: Request validation and security tokens
- **SSL/TLS Support**: Configurable certificate verification
- **Permission Control**: ACL integration with role-based access
- **Encrypted Payloads**: All data encrypted during transmission

### ⚡ **Advanced Features**
- **Lock System**: Prevents concurrent sync operations
- **Progress Logging**: Real-time sync progress tracking with detailed logs
- **Async Execution**: Background processing for large synchronization jobs
- **Asset Detection**: Automatic discovery and sync of linked assets
- **Retry Logic**: Automatic retry mechanism for failed requests
- **Error Handling**: Comprehensive error reporting and recovery

## 🔧 Configuration

### Basic Setup

Configure Sync in your `config/config.php`:

```php
<?php
return [
    // ... other config options
    
    'sync' => [
        'syncKey' => 'your-shared-secret-key'
    ]
];
```

### Required Configuration

**Sync Key** (Required on both instances):
```php
'sync' => [
    'syncKey' => 'your-secure-shared-secret'
]
```

⚠️ **Important**: The same `syncKey` must be configured on both source and target instances for successful synchronization.

### Sync Targets Configuration

Configure sync targets through the admin interface at `/sync/settings` or programmatically:

```php
// Example target configuration
$targets = [
    [
        'name' => 'Production Server',
        'uri' => 'https://production.example.com',
        'syncKey' => 'production-sync-key',
        'modes' => [
            'push' => true,    // Allow pushing to this target
            'pull' => false    // Disable pulling from this target
        ]
    ],
    [
        'name' => 'Staging Server',
        'uri' => 'https://staging.example.com',
        'syncKey' => 'staging-sync-key',
        'modes' => [
            'push' => true,
            'pull' => true
        ]
    ]
];

$app->dataStorage->setKey('cockpit', 'sync.targets', $targets);
```

### SSL Configuration

For production environments, configure SSL verification:

```php
// In your sync job configuration
$client = new GuzzleHttp\Client([
    'base_uri' => trim($target['uri'], '/') . '/',
    'verify' => true,  // Enable SSL verification
    'timeout' => 30,
    'connect_timeout' => 10
]);
```

## 🎯 Built-in Sync Jobs

### Content Synchronization

**Supports**: Collections, singletons, trees, content models, locales

```php
// Content sync settings
$contentSettings = [
    'syncAll' => false,           // Sync all models and data
    'locales' => true,           // Sync locale configurations
    'models' => ['blog', 'page'], // Specific models to sync
    'data' => ['blog'],          // Models to sync data for
    'mirror' => false            // Complete data replacement
];
```

### Asset Synchronization

**Supports**: Files, folders, metadata, automatic file download/upload

```php
// Assets are automatically included in content sync
// Or can be synced independently
$assetSettings = [
    'syncAll' => true  // Sync all assets and folders
];
```

### System Synchronization

**Supports**: User roles, permissions, ACL configurations

```php
// User roles sync settings
$roleSettings = [
    'syncAll' => false,
    'roles' => ['editor', 'author']  // Specific roles to sync
];
```

### Layout Synchronization

**Supports**: Layout definitions, component configurations

```php
// Layout sync (typically syncAll)
$layoutSettings = [
    'syncAll' => true
];
```

### Translation Projects (Lokalize)

**Supports**: Translation projects, keys, values, progress tracking

```php
// Lokalize sync settings
$lokalizeSettings = [
    'syncAll' => true,
    'projects' => ['website', 'app']  // Specific projects
];
```

## 🔨 Creating Custom Sync Jobs

### Basic Sync Job Structure

Create a new sync job by defining a configuration file.

```php
<?php
// addons/YourAddon/SyncJobs/custom/config.php

use GuzzleHttp\Client;

return [
    // Basic job information
    'name' => 'custom',
    'label' => 'Custom Data',
    'icon' => 'youraddon:icon.svg',
    
    // Configuration fields for admin UI (boolean flags pattern)
    'fields' => [
        [
            'name' => 'data',
            'type' => 'boolean',
            'info' => 'Sync main data'
        ],
        [
            'name' => 'settings',
            'type' => 'boolean',
            'info' => 'Sync configuration settings'
        ],
        [
            'name' => 'metadata',
            'type' => 'boolean',
            'info' => 'Sync metadata'
        ]
    ],
    
    // Push operation (send data to target)
    'sync:push' => function(array $settings = [], ?array $target = null, ?Client $client = null) {
        
        // Merge with defaults (Pages pattern)
        $settings = array_merge([
            'syncAll' => true,
            'data' => false,
            'settings' => false,
            'metadata' => false,
        ], $settings);
        
        // Extract settings for easier access
        extract($settings);
        
        // If syncAll is true, enable all components
        if ($syncAll) {
            $data = true;
            $settings = true;
            $metadata = true;
        }
        
        // Sync settings first (lightweight)
        if ($settings) {
            $payload = [
                'settings' => $this->module('custom')->settings()
            ];
            
            $client->request('POST', 'api/sync/job', [
                'json' => [
                    'job' => 'custom',
                    'mode' => 'push',
                    'payload' => $this->helper('jwt')->encode($payload, $target['syncKey'])
                ]
            ]);
        }
        
        // Sync metadata
        if ($metadata) {
            $payload = [
                'metadata' => $this->dataStorage->find('custom/metadata')->toArray()
            ];
            
            $client->request('POST', 'api/sync/job', [
                'json' => [
                    'job' => 'custom',
                    'mode' => 'push',
                    'payload' => $this->helper('jwt')->encode($payload, $target['syncKey'])
                ]
            ]);
        }
        
        // Sync main data (with batching for large datasets)
        if ($data) {
            $run = 0;
            $limit = 20;
            
            while (true) {
                $items = $this->dataStorage->find('custom/data', [
                    'skip' => $run * $limit,
                    'limit' => $limit
                ])->toArray();
                
                $payload = [
                    'run' => $run,
                    'data' => $items
                ];
                
                if (count($items)) {
                    $client->request('POST', 'api/sync/job', [
                        'json' => [
                            'job' => 'custom',
                            'mode' => 'push',
                            'payload' => $this->helper('jwt')->encode($payload, $target['syncKey'])
                        ]
                    ]);
                }
                
                // Break if no more items
                if (!count($items)) {
                    break;
                }
                
                $run += 1;
            }
        }
    },
    
    // Handle incoming push requests (Pages pattern)
    'on:push' => function($payload = null) {
        
        // Handle settings sync
        if (isset($payload['settings'])) {
            $settings = $payload['settings'];
            $this->trigger('custom.settings.save', [&$settings]);
            $this->dataStorage->setKey('custom/options', 'settings', $settings);
        }
        
        // Handle metadata sync
        if (isset($payload['metadata'])) {
            foreach ($payload['metadata'] as $meta) {
                $this->dataStorage->remove('custom/metadata', ['_id' => $meta['_id']]);
                $this->dataStorage->insert('custom/metadata', $meta);
            }
        }
        
        // Handle main data sync
        if (isset($payload['data'])) {
            
            // On first batch (run = 0), optionally clear existing data for mirror mode
            if (isset($payload['run']) && $payload['run'] === 0) {
                // Uncomment for mirror mode:
                // $this->dataStorage->dropCollection('custom/data');
            }
            
            foreach ($payload['data'] as $item) {
                $this->dataStorage->insert('custom/data', $item);
            }
        }
        
        return ['success' => true];
    },
    
    // Pull operation (request data from target) - Pages pattern
    'sync:pull' => function(array $settings = [], ?array $target = null, ?Client $client = null) {
        
        // Merge with defaults
        $settings = array_merge([
            'syncAll' => true,
            'data' => false,
            'settings' => false,
            'metadata' => false,
        ], $settings);
        
        extract($settings);
        
        // Enable all if syncAll is true
        if ($syncAll) {
            $data = true;
            $settings = true;
            $metadata = true;
        }
        
        // Pull settings
        if ($settings) {
            $response = $client->request('POST', 'api/sync/job', [
                'json' => [
                    'job' => 'custom',
                    'mode' => 'pull',
                    'payload' => $this->helper('jwt')->encode([
                        'settings' => true
                    ], $target['syncKey'])
                ]
            ]);
            
            $payload = json_decode($response->getBody()->getContents(), true);
            
            if (isset($payload['settings'])) {
                $settings = $payload['settings'];
                $this->trigger('custom.settings.save', [&$settings]);
                $this->dataStorage->setKey('custom/options', 'settings', $settings);
            }
        }
        
        // Pull metadata
        if ($metadata) {
            $response = $client->request('POST', 'api/sync/job', [
                'json' => [
                    'job' => 'custom',
                    'mode' => 'pull',
                    'payload' => $this->helper('jwt')->encode([
                        'metadata' => true
                    ], $target['syncKey'])
                ]
            ]);
            
            $payload = json_decode($response->getBody()->getContents(), true);
            
            if (isset($payload['metadata'])) {
                foreach ($payload['metadata'] as $meta) {
                    $this->dataStorage->remove('custom/metadata', ['_id' => $meta['_id']]);
                    $this->dataStorage->insert('custom/metadata', $meta);
                }
            }
        }
        
        // Pull main data with batching
        if ($data) {
            $run = 0;
            // Optionally clear collection for mirror mode
            // $this->dataStorage->dropCollection('custom/data');
            
            while (true) {
                $response = $client->request('POST', 'api/sync/job', [
                    'json' => [
                        'job' => 'custom',
                        'mode' => 'pull',
                        'payload' => $this->helper('jwt')->encode([
                            'data' => true,
                            'run' => $run
                        ], $target['syncKey'])
                    ]
                ]);
                
                $payload = json_decode($response->getBody()->getContents(), true);
                
                if (!isset($payload['data']) || !count($payload['data'])) {
                    break;
                }
                
                foreach ($payload['data'] as $item) {
                    $this->dataStorage->insert('custom/data', $item);
                }
                
                $run += 1;
            }
        }
    },
    
    // Handle incoming pull requests (Pages pattern)
    'on:pull' => function($payload = null) {
        
        // Handle settings request
        if (isset($payload['settings']) && $payload['settings']) {
            return ['settings' => $this->module('custom')->settings()];
        }
        
        // Handle metadata request
        if (isset($payload['metadata']) && $payload['metadata']) {
            return ['metadata' => $this->dataStorage->find('custom/metadata')->toArray()];
        }
        
        // Handle data request with batching
        if (isset($payload['data']) && $payload['data']) {
            $run = $payload['run'] ?? 0;
            $limit = 20;
            
            $items = $this->dataStorage->find('custom/data', [
                'skip' => $run * $limit,
                'limit' => $limit
            ])->toArray();
            
            return ['data' => $items];
        }
        
        return ['success' => false];
    }
];
```

### Individual Item Sync

For real-time synchronization of individual items (based on Pages implementation):

```php
<?php
// Add individual item sync handlers

return [
    // ... existing configuration
    
    // Push single item
    'sync:push:item' => function(array $payload, ?array $target = null, ?Client $client = null) {
        
        // Refresh item data before sync
        $payload['item'] = $this->dataStorage->findOne('custom/data', ['_id' => $payload['item']['_id']]);
        
        $response = $client->request('POST', 'api/sync/item', [
            'json' => [
                'job' => 'custom',
                'mode' => 'push',
                'payload' => $this->helper('jwt')->encode($payload, $target['syncKey'])
            ]
        ]);
        
        // Optional: Sync linked assets if specified
        if (isset($payload['syncAssets']) && $payload['syncAssets']) {
            $assets = $this->helper('sync')->getLinkedAssets($payload['item']);
            
            if (count($assets)) {
                $client->request('POST', 'api/sync/job', [
                    'json' => [
                        'job' => 'assets',
                        'mode' => 'push',
                        'payload' => $this->helper('jwt')->encode([
                            'run' => 0,
                            'uploads' => $this->fileStorage->getURL('uploads://'),
                            'assets' => $assets
                        ], $target['syncKey'])
                    ]
                ]);
            }
        }
        
        return json_decode($response->getBody()->getContents(), true);
    },
    
    // Handle single item push
    'on:push:item' => function(array $payload) {
        
        if (!isset($payload['item'])) {
            return ['success' => false];
        }
        
        // Validate parent relationships (for hierarchical data)
        if (isset($payload['item']['_pid']) && $payload['item']['_pid']) {
            $parent = $this->dataStorage->findOne('custom/data', ['_id' => $payload['item']['_pid']]);
            
            if (!$parent) {
                return ['success' => false, 'message' => 'Parent does not exist'];
            }
        }
        
        // Remove existing and insert updated
        $this->dataStorage->remove('custom/data', ['_id' => $payload['item']['_id']]);
        $this->dataStorage->insert('custom/data', $payload['item']);
        
        // Optional: Update related data (like routes, caches, etc.)
        // $this->helper('custom')->updateRelatedData($payload['item']['_id']);
        
        return ['success' => true];
    },
    
    // Pull single item
    'sync:pull:item' => function(array $payload, ?array $target = null, ?Client $client = null) {
        
        $response = $client->request('POST', 'api/sync/item', [
            'json' => [
                'job' => 'custom',
                'mode' => 'pull',
                'payload' => $this->helper('jwt')->encode($payload, $target['syncKey'])
            ]
        ]);
        
        $response = json_decode($response->getBody()->getContents(), true);
        
        if (!isset($response['item'])) {
            return ['success' => false, 'message' => 'Item not found'];
        }
        
        // Validate parent relationships
        if (isset($response['item']['_pid']) && $response['item']['_pid']) {
            $parent = $this->dataStorage->findOne('custom/data', ['_id' => $response['item']['_pid']]);
            
            if (!$parent) {
                return ['success' => false, 'message' => 'Parent does not exist'];
            }
        }
        
        // Update local item
        $this->dataStorage->remove('custom/data', ['_id' => $response['item']['_id']]);
        $this->dataStorage->insert('custom/data', $response['item']);
        
        // Optional: Update related data
        // $this->helper('custom')->updateRelatedData($response['item']['_id']);
        
        // Handle linked assets if requested
        if (isset($payload['syncAssets']) && $response['item']) {
            $assets = $this->helper('sync')->getLinkedAssets($response['item']);
            
            $streamContext = stream_context_create([
                'ssl' => [
                    'verify_peer' => false,
                    'verify_peer_name' => false,
                ],
            ]);
            
            foreach ($assets as $asset) {
                $path = trim($asset['path'], '/');
                $url = "{$response['uploads']}/{$path}";
                
                if (!$this->fileStorage->fileExists("uploads://{$path}")) {
                    // Check if resource exists
                    $headers = get_headers($url, 1, $streamContext);
                    
                    if ($headers === false || !str_contains($headers[0], ' 200')) {
                        continue;
                    }
                    
                    $stream = fopen($url, 'rb', false, $streamContext);
                    
                    if ($stream === false) {
                        continue;
                    }
                    
                    $this->fileStorage->writeStream("uploads://{$path}", $stream, ['mimetype' => $asset['mime']]);
                    
                    if (is_resource($stream)) {
                        fclose($stream);
                    }
                }
                
                $this->dataStorage->remove('assets', ['_id' => $asset['_id']]);
                $this->dataStorage->insert('assets', $asset);
            }
        }
        
        return ['success' => true, 'item' => $response['item']];
    },
    
    // Handle single item pull request
    'on:pull:item' => function(array $payload) {
        
        $item = $this->dataStorage->findOne('custom/data', ['_id' => $payload['item']['_id']]);
        
        return [
            'item' => $item,
            'uploads' => $this->fileStorage->getURL('uploads://')
        ];
    }
];
```

### Registering Custom Sync Jobs

Register your custom sync job in your addon's bootstrap:

```php
<?php
// addons/YourAddon/bootstrap.php

$this->on('sync.jobs', function($jobs) {
    
    // Add your custom sync job
    $jobs['custom'] = include(__DIR__ . '/SyncJobs/custom/config.php');
    
    // Add multiple jobs
    $jobs['custom:categories'] = include(__DIR__ . '/SyncJobs/categories/config.php');
    $jobs['custom:settings'] = include(__DIR__ . '/SyncJobs/settings/config.php');
});
```

### Advanced Sync Job Features

#### Error Handling and Retry Logic

```php
'sync:push' => function(array $settings = [], ?array $target = null, ?Client $client = null) {
    
    try {
        
        // Use retry helper for robust operations
        $this->helper('utils')->retry(3, function() use($client, $payload, $target) {
            
            $client->request('POST', 'api/sync/job', [
                'json' => [
                    'job' => 'custom',
                    'mode' => 'push',
                    'payload' => $this->helper('jwt')->encode($payload, $target['syncKey'])
                ],
                'timeout' => 60
            ]);
            
        }, 2); // 2 second delay between retries
        
    } catch (\Throwable $e) {
        
        $this->helper('sync')->log("Error syncing custom data: " . $e->getMessage());
        throw $e; // Re-throw to stop sync process
    }
}
```

#### Progress Tracking

```php
'sync:push' => function(array $settings = [], ?array $target = null, ?Client $client = null) {
    
    $totalItems = $this->dataStorage->count('custom/data');
    $processed = 0;
    
    $this->helper('sync')->log("Starting sync of {$totalItems} items...");
    
    // Process in batches
    $batches = 0;
    $batchSize = 20;
    
    while ($processed < $totalItems) {
        
        $items = $this->dataStorage->find('custom/data', [
            'skip' => $batches * $batchSize,
            'limit' => $batchSize
        ])->toArray();
        
        if (!count($items)) break;
        
        // Sync batch
        // ... sync logic here
        
        $processed += count($items);
        $percentage = round(($processed / $totalItems) * 100);
        
        $this->helper('sync')->log("Progress: {$processed}/{$totalItems} ({$percentage}%)");
        
        $batches++;
    }
    
    $this->helper('sync')->log("Custom data sync completed: {$processed} items");
}
```

#### Asset Dependency Handling

```php
'sync:push' => function(array $settings = [], ?array $target = null, ?Client $client = null) {
    
    $items = $this->dataStorage->find('custom/data')->toArray();
    
    foreach ($items as $item) {
        
        // Extract linked assets
        $linkedAssets = $this->helper('sync')->getLinkedAssets($item);
        
        if (count($linkedAssets)) {
            
            // Sync assets first
            $client->request('POST', 'api/sync/job', [
                'json' => [
                    'job' => 'assets',
                    'mode' => 'push',
                    'payload' => $this->helper('jwt')->encode([
                        'run' => 0,
                        'uploads' => $this->fileStorage->getURL('uploads://'),
                        'assets' => $linkedAssets
                    ], $target['syncKey'])
                ]
            ]);
        }
        
        // Then sync the item
        // ... sync item logic
    }
}
```

## 🔧 API Endpoints

### Manual Sync Execution

```php
// Trigger sync programmatically
$syncHelper = $app->helper('sync');

$target = [
    'uri' => 'https://target.example.com',
    'mode' => 'push',
    'syncKey' => 'shared-secret'
];

$jobs = [
    [
        'name' => 'content',
        'syncSettings' => [
            'syncAll' => false,
            'models' => ['blog'],
            'data' => ['blog']
        ]
    ]
];

$syncHelper->run($target, $jobs);
```

### Individual Item Sync

```php
// Sync individual item
$result = $app->helper('sync')->syncItem('content', $target, [
    'model' => ['name' => 'blog', 'type' => 'collection'],
    'item' => ['_id' => 'item-id-here'],
    'syncAssets' => true
]);
```

## 🐛 Troubleshooting

### Common Issues

**❌ "Target not available"**
- Verify target URL is accessible
- Check network connectivity between instances
- Ensure target instance has Sync addon enabled
- Verify SSL/TLS configuration

**❌ "Decoding payload failed"**
- Ensure both instances use the same `syncKey`
- Check for special characters in sync key
- Verify JWT encoding/decoding is working

**❌ "Sync job already running"**
- Another sync process is in progress
- Check for stale lock files in `/tmp` directory
- Use "Reset" function in admin interface if needed

**❌ "Permission denied"**
- Verify user has `sync/manage` permission
- Check ACL configuration on both instances
- Ensure API endpoints are accessible

### Debug Mode

Enable detailed logging for troubleshooting:

```php
// Check sync logs
$syncHelper = $app->helper('sync');
$logContent = $syncHelper->log();
echo $logContent;

// Check if sync is running
if ($syncHelper->isSyncRunning()) {
    echo "Sync is currently running";
}
```

### Performance Optimization

1. **Batch Size**: Adjust batch sizes for large datasets
2. **Network Timeout**: Increase timeout for slow connections
3. **Asset Sync**: Disable asset sync for content-only operations
4. **Selective Sync**: Use specific model/data filters instead of `syncAll`
5. **Async Processing**: Use background processing for large syncs

## 📄 License

This is a commercial addon for Cockpit CMS. See [LICENSE.md](LICENSE.md) for full terms.

## 🙏 Credits

Sync is developed by [Agentejo](https://agentejo.com) as part of the Cockpit CMS Pro ecosystem.

---

**Ready to synchronize?** Configure your sync keys and start building robust multi-instance workflows today!