# Detektivo

[![Commercial](https://img.shields.io/badge/License-Commercial-red.svg)](LICENSE.md)
[![Cockpit CMS](https://img.shields.io/badge/Cockpit%20CMS-Addon-blue.svg)](https://getcockpit.com)
[![Search Engine](https://img.shields.io/badge/Engine-Meilisearch%20%7C%20IndexLite-green.svg)](#supported-backends)

> **Advanced search index management and full-text search for Cockpit CMS**

Detektivo is a commercial addon that provides powerful search indexing capabilities for Cockpit CMS. It enables you to create custom search indexes for your content, pages, and any other data, with support for multiple search backends including Meilisearch and IndexLite.

## ✨ Features

### 🔍 **Flexible Search Indexing**
- **Multiple Index Types**: Content collections, Pages, and custom indexes
- **Field Selection**: Choose exactly which fields to index for optimal performance
- **Real-time Updates**: Automatic reindexing when content changes
- **Batch Processing**: Efficient background indexing for large datasets

### 🌐 **Multi-language Support**
- **Locale-aware Indexing**: Index content in multiple languages
- **Localized Search**: Search within specific locales or across all languages
- **Automatic Locale Detection**: Smart handling of i18n fields

### 🔧 **Advanced Configuration**
- **Custom Field Mapping**: Map content fields to search fields
- **Population Control**: Configure content linking depth before indexing
- **Filter Options**: Include/exclude content based on custom criteria
- **Metadata Enrichment**: Add computed fields and custom data

### 📊 **Management Interface**
- **Visual Index Management**: Create, edit, and monitor indexes via admin UI
- **Real-time Status**: Track indexing progress and health
- **Search Testing**: Built-in search interface for testing and debugging
- **Audit Logging**: Complete indexing history and performance metrics

### 🚀 **APIs & Integration**
- **REST API**: Full RESTful search API with authentication
- **GraphQL Support**: Native GraphQL queries and mutations
- **Admin Integration**: Seamless integration with Cockpit admin interface
- **Event System**: Hooks for custom indexing logic

## 🏗️ Supported Backends

Detektivo works with multiple search backends:

- **[Meilisearch](https://www.meilisearch.com/)** - Ultra-fast, typo-tolerant search engine
- **IndexLite** - Built-in lightweight search (SQLite-based)
- **Extensible** - Plugin architecture for additional backends

## 🚀 Quick Start

### 1. Installation

Detektivo is a commercial addon included with Cockpit CMS Pro licenses.

### 2. Create Your First Index

1. Navigate to **Detektivo** in the admin sidebar
2. Click **"Create Index"**
3. Choose an index type:
   - **Content Collection** - Index specific content models
   - **Pages** - Index your website pages
   - **Custom Index** - Build completely custom indexes

### 3. Configure Index Fields

```php
// Example: Content Collection Index
$index = [
    'name' => 'articles',
    'type' => 'content:collection',
    'fields' => ['title', 'content', 'tags', 'author'],
    'meta' => [
        'model' => 'articles',
        'populate' => 1,
        'locale' => true
    ]
];
```

### 4. Search Your Content

```javascript
// Frontend search example
fetch('/api/detektivo/search/articles?q=cockpit cms&limit=10')
    .then(response => response.json())
    .then(results => {
        console.log(`Found ${results.estimatedTotalHits} articles`);
        results.hits.forEach(article => {
            console.log(article.title);
        });
    });
```

## 📋 Index Types

### Content Collections

Index any content model with full control over fields and localization:

```php
return [
    'name' => 'content:collection',
    'label' => 'Collection',
    'meta' => [
        'model' => 'blog_posts',      // Content model name
        'populate' => 2,              // Link population depth
        'locale' => true              // Include all locales
    ],
    'fields' => ['title', 'content', 'author', 'tags', 'published']
];
```

**Features:**
- ✅ Automatic content model discovery
- ✅ Configurable field selection
- ✅ Multi-locale support
- ✅ Linked content population
- ✅ Real-time updates via events

### Pages

Index your website pages with full content and metadata:

```php
return [
    'name' => 'pages',
    'label' => 'Pages',
    'meta' => [
        'populate' => 1               // Content population depth
    ],
    'fields' => ['title', 'content', 'meta', 'route', 'locale']
];
```

**Features:**
- ✅ Full page content indexing
- ✅ Route and URL mapping
- ✅ Meta data inclusion
- ✅ Nested page support
- ✅ Locale-specific routing

### Custom Indexes

Build completely custom indexes with your own data sources:

```php
return [
    'name' => 'custom',
    'label' => 'Custom Index',
    'fields' => function($meta) {
        // Return dynamic field list based on configuration
        return ['id', 'title', 'description', 'data'];
    },
    'index' => function($index, $idx) {
        // Custom indexing logic
        $documents = $this->getCustomData($index['meta']);
        $idx->addDocuments($documents);
    }
];
```

## 🔍 Search API

### REST API

**Search Endpoint:**
```
GET /api/detektivo/search/{index}
```

**Parameters:**
- `q` (required) - Search query string
- `fields` (optional) - Comma-separated fields to return
- `limit` (optional) - Maximum results (default: 50)
- `offset` (optional) - Result offset for pagination
- `filter` (optional) - Additional filters
- `facets` (optional) - Comma-separated list or JSON array of facet fields (multi-facet)
- `facet_limit` (optional) - Max facet values per field (default: 20)
- `facet_offset` (optional) - Offset for facet pagination per field (default: 0)
- `boosts` (optional) - JSON object of field boosts (e.g. `{ "title": 2.0, "content": 1.0 }`)

**Fuzzy Search Parameters:**
- `fuzzy` (optional) - Enable fuzzy search/typo tolerance (boolean) - **Both backends**
- `highlights` (optional) - Enable search term highlighting (boolean) - **Both backends**
- `attributes_to_highlight` (optional) - Comma-separated fields to highlight - **Both backends**

**IndexLite Backend Only:**
- `fuzzy_algorithm` (optional) - Algorithm: `levenshtein`, `jaro_winkler`, `trigram`, `soundex`, `hybrid`, `fts5`
- `fuzzy_threshold` (optional) - Distance threshold (0.0-1.0)
- `fuzzy_min_score` (optional) - Minimum match score (0.0-1.0)

**Meilisearch Backend Only:**
- `typo_tolerance` (optional) - Enable Meilisearch typo tolerance (boolean)
- `ranking_score_threshold` (optional) - Minimum ranking score (0.0-1.0)

Note on facets (Meilisearch): facet attributes must be configured as `filterableAttributes`. The adapter now auto‑ensures requested facet fields are filterable before running the query.

**Basic Example:**
```bash
curl "https://yoursite.com/api/detektivo/search/articles?q=cockpit+cms&fields=title,content&limit=5"
```

**Multi‑Facet Example (both backends):**
```bash
curl "https://yoursite.com/api/detektivo/search/articles\
  ?q=phone\
  &facets=category,brand\
  &facet_limit=10\
  &boosts={\"title\":2.5}"
```

Response contains a unified `facets` object:
```json
{
  "hits": [ /* ... */ ],
  "estimatedTotalHits": 12,
  "limit": 10,
  "offset": 0,
  "processingTimeMs": 8,
  "facets": {
    "category": [
      { "value": "Electronics", "count": 7 },
      { "value": "Computers",   "count": 3 }
    ],
    "brand": [
      { "value": "Samsung",  "count": 4 },
      { "value": "Apple",    "count": 3 }
    ]
  }
}
```

Notes:
- Facets are sorted by `count` descending and respect `facet_limit`/`facet_offset` per field.
- `boosts` affect ranking on IndexLite via bm25 column weights. On Meilisearch, boosting typically requires index ranking rules; the parameter is accepted for parity but not auto‑applied.

**Fuzzy Search Examples:**

*Basic fuzzy search (works with both backends):*
```bash
curl "https://yoursite.com/api/detektivo/search/articles?q=cockpitt&fuzzy=true"
```

*IndexLite backend with specific algorithm:*
```bash
curl "https://yoursite.com/api/detektivo/search/articles?q=cockpitt&fuzzy=true&fuzzy_algorithm=levenshtein&fuzzy_threshold=0.8"
```

*IndexLite backend with advanced scoring:*
```bash
curl "https://yoursite.com/api/detektivo/search/articles?q=cokcpit&fuzzy=true&fuzzy_algorithm=jaro_winkler&fuzzy_min_score=0.7"
```

*Meilisearch backend with typo tolerance and highlighting:*
```bash
curl "https://yoursite.com/api/detektivo/search/articles?q=cockpitt&typo_tolerance=true&highlights=true&attributes_to_highlight=title,content"
```

**Response:**
```json
{
    "hits": [
        {
            "id": "article_123",
            "title": "Getting Started with Cockpit CMS",
            "content": "Cockpit CMS is a powerful...",
            "_score": 0.95
        }
    ],
    "estimatedTotalHits": 42,
    "limit": 5,
    "offset": 0,
    "processingTimeMs": 12
}
```

### GraphQL

**Basic Search:**
```graphql
query SearchArticles($query: String!, $limit: Int) {
    detektivoSearch(index: "articles", q: $query, limit: $limit) {
        hits {
            id
            title
            content
            _score
        }
        estimatedTotalHits
        processingTimeMs
    }
}
```

**Fuzzy Search with Highlighting:**
```graphql
query FuzzySearchArticles(
    $query: String!,
    $fuzzy: Boolean,
    $algorithm: String,
    $highlights: Boolean
) {
    detektivoSearch(
        index: "articles",
        q: $query,
        fuzzy: $fuzzy,
        fuzzyAlgorithm: $algorithm,
        highlights: $highlights,
        attributesToHighlight: ["title", "content"]
    ) {
        hits {
            id
            title
            content
            _score
            _formatted
        }
        estimatedTotalHits
        processingTimeMs
    }
}
```

**Variables:**
```json
{
    "query": "cockpitt cms",
    "fuzzy": true,
    "algorithm": "levenshtein",
    "highlights": true
}
```

## 🔮 Fuzzy Search & Typo Tolerance

Detektivo provides advanced fuzzy search capabilities to handle typos, spelling variations, and approximate matches across different search backends.

### IndexLite Backend Fuzzy Algorithms

**Available Algorithms** (IndexLite/SQLite only):

1. **`levenshtein`** - Edit distance (insertions, deletions, substitutions)
   - Best for: General typo correction
   - Example: "cockpit" matches "cockpitt", "cokpit", "cockpig"

2. **`jaro_winkler`** - String similarity with prefix weighting
   - Best for: Names and proper nouns
   - Example: "Smith" matches "Smyth", "Smit"

3. **`trigram`** - Three-character substring matching
   - Best for: Partial word matches
   - Example: "development" matches "develop", "velop"

4. **`soundex`** - Phonetic matching algorithm
   - Best for: Words that sound similar
   - Example: "Jackson" matches "Jakson", "Jaxon"

5. **`hybrid`** - Combined scoring from multiple algorithms
   - Best for: Maximum recall with relevance ranking

6. **`fts5`** - SQLite full-text search with prefix matching
   - Best for: Performance with large datasets

### Meilisearch Backend Typo Tolerance

Meilisearch provides built-in typo tolerance with configurable settings. **Note**: `fuzzy_algorithm`, `fuzzy_threshold`, and `fuzzy_min_score` parameters are ignored by Meilisearch - use `typo_tolerance` and `ranking_score_threshold` instead.

```php
// Advanced typo tolerance configuration
$params = [
    'typoTolerance' => true,
    'rankingScoreThreshold' => 0.8,  // Minimum relevance score
    'highlights' => true,             // Highlight matched terms
    'attributesToHighlight' => ['title', 'content']
];
```

### Usage Examples

**Basic Fuzzy Search:**
```javascript
// Enable fuzzy search for typo tolerance
const results = await fetch('/api/detektivo/search/articles?q=cockpitt&fuzzy=true');
```

**Algorithm-Specific Search:**
```javascript
// Use Levenshtein for exact typo correction
const results = await fetch('/api/detektivo/search/articles', {
    method: 'GET',
    body: new URLSearchParams({
        q: 'developmnt',
        fuzzy: true,
        fuzzy_algorithm: 'levenshtein',
        fuzzy_threshold: 0.8,
        fuzzy_min_score: 0.6
    })
});
```

**Highlighted Results:**
```javascript
// Get search results with highlighted matches
const results = await fetch('/api/detektivo/search/articles', {
    method: 'GET', 
    body: new URLSearchParams({
        q: 'javascrpt tutorial',
        highlights: true,
        attributes_to_highlight: 'title,content',
        typo_tolerance: true
    })
});

// Response includes _formatted field with highlights
console.log(results.hits[0]._formatted.title);
// Output: "<mark>JavaScript</mark> <mark>Tutorial</mark>"
```

### Performance Tuning

**Threshold Configuration:**
```javascript
// Fine-tune fuzzy matching sensitivity
const params = {
    fuzzy_threshold: 0.85,    // Higher = stricter matching
    fuzzy_min_score: 0.7,     // Minimum relevance score
    ranking_score_threshold: 0.8  // Meilisearch only
};
```

**Algorithm Selection Guide:**
- **High accuracy needed**: `levenshtein` with threshold 0.8+
- **Performance critical**: `fts5` or `trigram`
- **Name/person search**: `jaro_winkler`
- **Maximum recall**: `hybrid` with lower thresholds
- **Sound-based matching**: `soundex`

### Frontend Integration

```javascript
// React hook with fuzzy search
const useFuzzySearch = (index, options = {}) => {
    const [query, setQuery] = useState('');
    const [results, setResults] = useState([]);
    
    const search = useCallback(async (searchQuery) => {
        const params = new URLSearchParams({
            q: searchQuery,
            fuzzy: true,
            fuzzy_algorithm: options.algorithm || 'levenshtein',
            fuzzy_threshold: options.threshold || 0.8,
            highlights: true,
            ...options.params
        });
        
        const response = await fetch(`/api/detektivo/search/${index}?${params}`);
        const data = await response.json();
        setResults(data.hits || []);
    }, [index, options]);
    
    return { query, setQuery, results, search };
};
```

## ⚙️ Configuration

### Backend Configuration

Configure your search backend in `config/config.php`:

```php
// Meilisearch configuration
'search' => [
    'type' => 'Meilisearch',
    'options' => [
        'host' => 'http://localhost:7700',
        'masterKey' => 'your-master-key'
    ]
],

// IndexLite configuration (default)
'search' => [
    'type' => 'IndexLite',
    'options' => [
        'database' => '#storage:search.db'
    ]
]
```

### Index Field Configuration

```php
// Define searchable fields
'fields' => [
    'title' => [
        'weight' => 100,        // Higher weight = more important
        'searchable' => true,   // Include in search
        'stored' => true        // Return in results
    ],
    'content' => [
        'weight' => 50,
        'searchable' => true,
        'stored' => false       // Don't return full content
    ],
    'tags' => [
        'weight' => 75,
        'searchable' => true,
        'faceted' => true       // Enable faceted search
    ]
]
```

## 🔧 Advanced Usage

### Custom Index Types

Create your own index types by adding configuration files:

```php
// /addons/YourAddon/IndexTypes/custom/config.php
return [
    'name' => 'custom:products',
    'label' => 'Product Catalog',
    'icon' => 'youraddon:assets/icons/product.svg',
    
    'meta' => [
        [
            'name' => 'category',
            'type' => 'select',
            'label' => 'Product Category',
            'options' => ['electronics', 'clothing', 'books']
        ]
    ],
    
    'fields' => function($meta) {
        return ['name', 'description', 'price', 'category', 'sku'];
    },
    
    'index' => function($index, $idx) {
        // Custom indexing logic
        $products = $this->getProductData($index['meta']);
        
        foreach ($products as $product) {
            $document = [
                'id' => $product['id'],
                'name' => $product['name'],
                'description' => $product['description'],
                'price' => (float) $product['price'],
                'category' => $product['category'],
                'sku' => $product['sku']
            ];
            
            $idx->addDocuments([$document]);
        }
    }
];
```

### Event Hooks

Hook into the indexing process for custom logic:

```php
// Register custom index types
$this->on('detektivo.collect.indextypes', function($types, $include) {
    $types['custom:products'] = $include(__DIR__.'/custom-products.php');
});

// Modify documents before indexing
$this->on('detektivo.index.document', function($document, $index) {
    // Add computed fields
    $document['search_boost'] = calculateSearchBoost($document);
    $document['indexed_at'] = time();
    
    return $document;
});

// Post-indexing cleanup
$this->on('detektivo.remove.index', function($indexName) {
    // Custom cleanup logic
    $this->clearCustomCache($indexName);
});
```

### Background Indexing

For large datasets, use the background indexer:

```php
// Start background indexing
$indexer = new \Detektivo\Helper\Indexer('large_index', $app);
$result = $indexer->start([
    'batch_size' => 100,
    'memory_limit' => '512M'
]);

// Check indexing status
if ($indexer->isIndexerRunning()) {
    $log = $indexer->getLog();
    echo "Status: " . end($log)['msg'];
}
```

### CLI Indexing

Use CLI commands for batch operations:

```bash
# List all configured indexes
./tower detektivo:list

# List indexes with available index types
./tower detektivo:list --types

# Check status of all indexes
./tower detektivo:status

# Check detailed status of a specific index
./tower detektivo:status articles

# Index all configured indexes
./tower detektivo:index --all

# Index a specific index
./tower detektivo:index --name=articles

# Clear and rebuild an index
./tower detektivo:index --name=articles --rebuild

# Clear all documents from an index
./tower detektivo:clear articles

# Clear without confirmation prompt
./tower detektivo:clear articles --force
```

**Available Commands:**

| Command | Description |
|---------|-------------|
| `detektivo:list` | List all configured indexes (`--types` to show available index types) |
| `detektivo:status` | Show index status (add index name for detailed view) |
| `detektivo:index` | Run indexer (`--all`, `--name=<index>`, `--rebuild`) |
| `detektivo:clear` | Clear documents from an index (`--force` to skip confirmation) |

## 🎨 Frontend Integration

### Vue.js Component

```vue
<template>
    <div class="search-component">
        <input 
            v-model="query" 
            @input="search"
            placeholder="Search..."
            class="search-input"
        />
        
        <div v-if="loading" class="loading">Searching...</div>
        
        <div v-else-if="results.length" class="results">
            <div 
                v-for="result in results" 
                :key="result.id"
                class="result-item"
            >
                <h3>{{ result.title }}</h3>
                <p>{{ result.excerpt }}</p>
                <span class="score">Score: {{ result._score }}</span>
            </div>
        </div>
        
        <div v-else-if="query" class="no-results">
            No results found for "{{ query }}"
        </div>
    </div>
</template>

<script>
export default {
    data() {
        return {
            query: '',
            results: [],
            loading: false
        }
    },
    
    methods: {
        async search() {
            if (!this.query.trim()) {
                this.results = [];
                return;
            }
            
            this.loading = true;
            
            try {
                const response = await fetch(
                    `/api/detektivo/search/articles?q=${encodeURIComponent(this.query)}&limit=10`
                );
                const data = await response.json();
                this.results = data.hits || [];
            } catch (error) {
                console.error('Search error:', error);
            } finally {
                this.loading = false;
            }
        }
    }
}
</script>
```

### React Hook

```javascript
import { useState, useEffect, useCallback } from 'react';

export function useDetektivoSearch(indexName, options = {}) {
    const [query, setQuery] = useState('');
    const [results, setResults] = useState([]);
    const [loading, setLoading] = useState(false);
    const [error, setError] = useState(null);
    
    const search = useCallback(async (searchQuery) => {
        if (!searchQuery.trim()) {
            setResults([]);
            return;
        }
        
        setLoading(true);
        setError(null);
        
        try {
            const params = new URLSearchParams({
                q: searchQuery,
                limit: options.limit || 10,
                ...options.params
            });
            
            const response = await fetch(
                `/api/detektivo/search/${indexName}?${params}`
            );
            
            if (!response.ok) {
                throw new Error(`Search failed: ${response.statusText}`);
            }
            
            const data = await response.json();
            setResults(data.hits || []);
        } catch (err) {
            setError(err.message);
            setResults([]);
        } finally {
            setLoading(false);
        }
    }, [indexName, options]);
    
    useEffect(() => {
        const debounceTimer = setTimeout(() => {
            search(query);
        }, 300);
        
        return () => clearTimeout(debounceTimer);
    }, [query, search]);
    
    return {
        query,
        setQuery,
        results,
        loading,
        error,
        search
    };
}
```

## 📊 Performance & Optimization

### Indexing Best Practices

1. **Field Selection**: Only index fields you actually search
2. **Population Limits**: Use minimal population levels to reduce memory usage
3. **Batch Size**: Adjust batch sizes based on content complexity
4. **Background Processing**: Use background indexer for large datasets

### Search Optimization

```php
// Optimize search performance
$searchParams = [
    'limit' => 20,                    // Reasonable page size
    'fields' => 'title,excerpt',      // Only needed fields
    'filter' => ['status' => 'published'], // Pre-filter results
    'facets' => ['category', 'tags']  // Enable faceted search
];
```

### Memory Management

```php
// Configure indexer for large datasets
$meta = [
    'batch_size' => 50,        // Smaller batches
    'memory_limit' => '256M',  // Reasonable memory limit
    'timeout' => 300           // 5 minute timeout
];

$this->module('detektivo')->runIndexer('large_index', $meta);
```

## 🔒 Security & Permissions

### API Authentication

```javascript
// Authenticated search requests
const searchWithAuth = async (query) => {
    const response = await fetch('/api/detektivo/search/articles', {
        method: 'GET',
        headers: {
            'Cockpit-Token': 'your-api-token',
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ q: query })
    });
    
    return response.json();
};
```

## 🐛 Troubleshooting

### Common Issues

**Search returns no results:**
- Verify index exists and has documents
- Check field configuration and searchable attributes
- Ensure search backend is running and accessible

**Performance issues:**
- Reduce indexed fields to essential ones only
- Use pagination for large result sets
- Consider search result caching


## 📄 License

This is a commercial addon for Cockpit CMS. See [LICENSE.md](LICENSE.md) for full terms.

## Credits

Detektivo is developed by [Agentejo](https://agentejo.com) as part of the Cockpit CMS ecosystem.

---

**Ready to supercharge your search?** Install Detektivo and start building powerful search experiences today!
**Multi‑Facet Search:**
```graphql
query SearchWithFacets(
  $q: String!,
  $facets: [String!],
  $facetLimit: Int,
  $facetOffset: Int
) {
  detektivoSearch(
    index: "articles",
    q: $q,
    facets: $facets,
    facetLimit: $facetLimit,
    facetOffset: $facetOffset
  ) {
    hits { id title }
    estimatedTotalHits
    facets
  }
}
```

Example Variables:
```json
{
  "q": "phone",
  "facets": ["category", "brand"],
  "facetLimit": 10
}
```
