Correlation Engine

🪙 Identity Engine: All the Evidence

• Tz-comparison bug fixed: the engine was iterating only the FIRST row of every feather when a time filter was active — a timezone-aware vs naive datetime comparison raised TypeError and aborted the per-row loop. Records seen jumped from 3,558 → 745,615 on the validation case.
• Log records no longer collapse to one identity: every SecurityLogs record used to be extracted as "Microsoft-Windows-Security-Auditing" (the event PROVIDER). The per-artifact mapping now prioritises real entities (User, ComputerName, NewProcessName, TargetUserName) before channel metadata.
• Artifact-aware mapping now actually fires: parsers rarely stamp an artifact column on each row, so the artifact-specific field-priority list never engaged. The engine now falls back to feather_metadata.artifact_type.
• Placeholder strings filtered: "N/A", "Unknown", "-", nil-GUIDs, and 25+ similar variants are now rejected as identities (case-folded match). They were bundling tens of thousands of unrelated records into one bogus bucket.

📊 Real "High vs Low" Labeling

• Identity engine used to tag every match High. Now matches with feather_count == 1 get "Low - single feather" — same convention as the time engine. The analyst's High view focuses on real cross-feather correlation.
• Path-aware composite key removed from primary grouping: each feather stores paths differently (BAM uses /device/harddiskvolume3/…, LNK uses c:/users/…, MFT uses 8.3 short paths). With path in the key, chrome had 10+ different composite keys and never correlated. Primary key is now name-only (canonical identity_grouping.identity_key) — same as the time engine. Cross-feather correlation works again.
• Validation impact: Execution Proof surfaces 2,856 cross-feather (High) matches; Account Logon went from 8 → 24 cross-feather; Anti-Forensics from 5 → 40; Lateral Movement from 16 → 40.

🚨 Impersonation Detection

• Path-based impersonation flag replaces what path-aware keying was supposed to provide. After a match is formed, the engine walks all records' path-like fields and classifies as TRUSTED (Program Files, Windows\System32, WinSxS, WindowsApps, ProgramData\Microsoft, BAM/SRUM /device/harddiskvolumeN/program files/…) or SUSPICIOUS (Temp, Downloads, Public, AppData\Local\Temp, $Recycle.Bin, removable-drive roots, network shares).
• A match with records in BOTH classifications gets match.semantic_data['impersonation_alert']. On the validation case: ~50 alerts per wing (≈ 0.05% rate) — each a real candidate (python.exe in venv + Program Files, Discord's official install + AppData copy, etc.).
• SUSPICIOUS wins on overlap: a path like c:\windows\temp\… won't double-count just because c:\windows is a trusted prefix.

📒 Engine Evidence Accounting

• Per-window drop ledger: named buckets (no_identity_field, normalize_failure, below_threshold_skipped, etc.) carry the first 3 sample identities so logs name what was lost, not just how many.
• Per-pipeline summary block: every correlate(…) call ends with records seen, high-conf emitted, low-conf emitted, no-identity drops, and timeless-feather records joined by identity. Every record either lands in a match or in a named drop bucket.
• low_confidence_review_mode is now ON by default. Below-threshold identity groups become Low-confidence matches instead of being silently dropped.

🪶 Timeless-Feather Identity Enrichment

• Feathers without per-row timestamps (AutoStartPrograms, MUICache, SystemServices, TypedPaths) no longer get a synthetic feather-generation timestamp stamped onto every row (which used to lie about when the artifact was observed).
• Instead, after time-windowed matches are formed by the timed feathers, the engine walks each match's identity and pulls matching records from every timeless feather, attaching them as supplementary evidence. Identity drives the join; the time anchor comes from the timed feathers.

📇 Consolidated Identity Registry

• New config/standard_fields/identities.json — single source of truth for every column the engines + EYE Agent should consult to extract or track an identity. 98 categories, 1,146 column synonyms covering app/process, file, hash, user, host/device, network, registry, service/task, event, email, browser, cloud (AWS / Azure / GCP), Windows internals (mutex, named pipe, COM CLSID, ETW provider, WMI consumer), certificate, container, and OS objects.
• Engine's timestamp-field and path-field lists now read directly from the JSON registries instead of hand-maintained Python lists. Adding a new parser column synonym is a JSON edit, not a code change.
• timestamps.json cleaned up: real MFT activity timestamps (si_creation_time, si_modification_time, si_mft_entry_change_time, etc.) moved out of the bookkeeping category where they were wrongly classified.

🧠 Semantic Mapping False-Positive Fixes

• default-data-exfiltration-pattern: was an OR rule across 3 wildcard conditions on different feathers — fired severity=high on any single LNK record. Now properly gated with _requires_multi_indicator=True, _min_indicators=2. The previously-dead multi-indicator fields are now actually honored.
• auth_failed_then_success: was logically impossible (AND on EventID == 4625 AND EventID == 4624 — a single record can equal only one value, rule never fired). Rewritten as OR with clearer semantic value.
• Wiper / Remote tool rules described specific tools (sdelete.exe, cipher.exe etc.) but had wildcard conditions — every Prefetch entry got tagged "Wiper Tool Run" at severity=high. Now use proper regex matching the actual tool names.
• Severity inflation fixes on exec_confirmed_run, pers_run_key, pers_service_install, pers_confirmed_runs — baseline-activity rules demoted from high/critical to info/low. The wing's weighted scoring layer escalates real threats via score thresholds.

⏱️ Time-Engine Accuracy Layer

• Method 0 schema-driven timestamp detection from feather_schemas.json. Feathers no longer rely on auto-detection that could pick integer cycle-counters (foreground_cycle_time) as Unix-epoch timestamps.
• Per-column WHERE-clause format binding. Heterogeneous-format feathers (amcache MM/DD/YYYY install_date + ISO parsed_at) used to silently return 0 rows because the engine used one global format. Now each column's WHERE parameters use that column's specific format.
• Multi-column timestamp fan-out for separate-column timestamp feathers (mft_usn: si_creation_time + si_modification_time + si_access_time + 6 more). A single row with timestamps in multiple windows becomes a virtual record in each. mft_usn went from 2 records returned to 913,809 virtual records.
• Year 1990–2100 plausibility filter on parsed timestamps drops NTFS / LNK placeholder dates (1601-01-01, 1980-01-01, 2000-01-01) before they spawn lonely virtual records.

📦 No Dropped Evidence

• Multi-timestamp fan-out: Prefetch's run_times JSON list (up to 8 historical executions per row) is expanded into one virtual correlation event per timestamp. Real cases see ~4× more execution events surfaced than the previous engine.
• Tolerant timestamp parser: Windows FILETIME, YYYYMMDD compact dates, trailing parser annotations like "2026-05-19 10:48:21 (Registry Key LastWrite)" all parse on the first try.
• Duplicates preserved as evidence: Dedup keys on the raw identity, not the normalized form. Chrome.exe and Chrome.dll no longer collapse into one row.
• 290+ identity-field synonyms recognized — MFT, BAM, USB, BrowserHistory, Shellbags, RecycleBin, Network_list, and UserProfiles records that previously had "no identity" now form matches.

🎯 Smarter Sub-Identity Grouping

• Chrome.exe / chrome.exe / Chrome.EXE / chrome.dll → one sub-identity (case + extension are noise).
• Chrome v1.0.exe ≠ Chrome v2.0.exe (versions are signal).
• Chrome (x86) ≠ Chrome (x64) (architectural qualifiers are signal).
• Each sub-identity exposes its full set of name variants in the UI.

🧰 One Source of Truth

• Field synonyms live in config/standard_fields/*.json. Adding a new parser column means editing JSON, not three Python files.
• Per-table metadata (primary timestamp column, multi-timestamp JSON columns, identity priority) lives in feather_schemas.json.
• Identity normalization unified in one module — engine, viewers, and semantic phase share one implementation.

⚡ Performance Foundation

• Thread-safe feather query cache (RLock-protected). Path to true parallel correlation is unblocked.
• Streaming reads via query_time_range_iter() — yields records in 1000-row batches, constant memory across feathers of any size.
• parallel_strategy config field ready for opt-in process-pool parallelism ('none' | 'threads' | 'processes').

📝 Better Feather Generation

• FeatherWriter — canonical write contract with WAL journal, explicit transactions, and executemany() batched inserts (5000-row batches). Expected 50–200× speedup on large imports.
• Schema metadata stamped into the feather DB itself; engine reads it instead of sniffing sample values.
• Multi-timestamp JSON columns declarable via one writer.declare_multi_timestamp_json(...) call — zero engine-side changes.

🩺 Diagnostics You Can Trust

• Per-window INFO line: records_in / no_identity / parse_cache_hits / below_threshold / matches_emitted / min_feathers. If matches drop, the log says exactly where.
• _last_window_correlation_stats exposes the same data programmatically for downstream tooling.
• low_confidence_review_mode + min_feathers_override let analysts dial correlation strictness per case.

⚡ Speed & Efficiency

The dual-engine architecture is optimized for performance. The Identity-Based engine processes large datasets with O(N log N) complexity and streaming mode, enabling analysis of millions of records without memory constraints. Investigators can correlate data as fast as possible to get answers when time is critical.

🎯 Accuracy & Focus

By automatically correlating artifacts across multiple sources (Prefetch, Registry, Event Logs, MFT, SRUM, and more), the engine helps investigators focus on the most needed details. Instead of manually searching through thousands of records, investigators can immediately see relationships and patterns that matter to their case.

🤝 Community-Driven

From the community, to the community. Crow Eye is built as an open-source solution by forensic investigators who understand the challenges of digital forensics. The correlation engine is actively developed with contributions from the forensic community, ensuring it evolves to meet real-world investigation needs.

1. Identity-Based Correlation Engine

Groups records by identity first, then creates temporal anchors. Optimized for large datasets (> 1,000 records) with O(N log N) performance and streaming support. Production Ready

2. Time-Based Correlation Engine

Utilizes an O(N log N) time-window scanning approach for systematic temporal analysis. Ideal for large datasets (> 1,000 records) requiring performance-critical analysis. Production Ready

Identity-Based Methodology

1. Identity Extraction: Normalizes application names, file paths, and hashes across all artifacts using semantic mappings
2. Identity Grouping: Groups all records by their normalized identity (e.g., all evidence related to "chrome.exe")
3. Temporal Clustering: Within each identity group, creates temporal anchors representing clusters of related events
4. Cross-Artifact Correlation: Links evidence from different artifacts (Prefetch, Registry, Event Logs) for the same identity
5. Streaming Output: Writes results directly to database to maintain constant memory usage
6. Confidence Scoring: Calculates scores based on identity strength, temporal proximity, and artifact diversity

Key Advantage: Its identity-centric approach ensures highly efficient tracking of specific applications or files across the entire forensic timeline, regardless of their timestamps, without loading all data into memory.

Time-Based Methodology

1. Time Range Detection: Automatically determines the overall time span of all artifacts.
2. Window Generation: Divides the entire time range into fixed, overlapping time windows.
3. Indexed Window Querying: For each window, efficiently queries all relevant records from all feathers using optimized database indexes.
4. In-Window Correlation: Correlates records within the current time window using comprehensive field-level matching and semantic rules.
5. Streaming Output: Writes correlated results directly to the database for constant memory usage.
6. Confidence Scoring: Calculates scores based on temporal proximity, field matches, and semantic similarity within each window.

Key Advantage: Offers a systematic and comprehensive temporal view of all activities within defined time windows, ideal for reconstructing events and analyzing activity patterns across the entire dataset.

engine/ - Core Correlation Engine

Purpose: Contains the core correlation logic, feather loading, scoring, and result management.

Key Files: 35 Python files

• correlation_engine.py - Main correlation engine
• identity_correlation_engine.py - Identity-based correlation
• time_based_engine.py - Time-based correlation
• engine_selector.py - Engine factory and selection
• weighted_scoring.py - Confidence score calculation

feather/ - Data Normalization

Purpose: Handles importing forensic artifact data from various sources and normalizing it into the feather format.

Key Files: 4 Python files + UI subdirectory

• __init__.py
• feather_builder.py - Main application entry point
• database.py - Database operations
• transformer.py - Data transformation pipeline
• ui/ - GUI components for Feather Builder

wings/ - Correlation Rules

Purpose: Defines the data models and validation logic for Wing configurations (correlation rules).

Key Files: 4 Python files in core/ + UI subdirectory

• core/__init__.py
• core/wing_model.py - Wing, FeatherSpec, CorrelationRules data models
• core/artifact_detector.py - Detect artifact types
• core/wing_validator.py - Validate wing configurations
• ui/ - GUI components for Wings Creator

config/ - Configuration Management

Purpose: Manages all configuration files (feathers, wings, pipelines) and semantic mappings.

Key Files: 20 Python files

• config_manager.py - Central configuration management
• semantic_mapping.py - Semantic field mappings
• pipeline_config.py - Pipeline configuration model
• integrated_configuration_manager.py - Integrated configuration
• case_specific_configuration_manager.py - Case-specific configuration

pipeline/ - Workflow Orchestration

Purpose: Orchestrates complete analysis workflows, including feather creation, wing execution, and report generation.

Key Files: 7 Python files

• pipeline_executor.py - Main pipeline execution
• pipeline_loader.py - Load pipeline configurations
• feather_auto_registration.py - Auto-register feathers
• discovery_service.py - Discover available configs
• database_connection_manager.py - Manage pipeline database connections

gui/ - User Interface

Purpose: Provides all GUI components for the correlation engine, including pipeline management, results visualization, and configuration editing.

Key Files: 35 Python files

• main_window.py - Main application window
• pipeline_management_tab.py - Pipeline creation/management
• correlation_results_view.py - Correlated results display
• timeline_widget.py - Timeline visualization
• settings_dialog.py - User settings and configurations

integration/ - Crow-Eye Bridge

Purpose: Integrates the correlation engine with the main Crow-Eye application, providing auto-generation features and default configurations.

Key Files: 21 Python files + default_wings/ subdirectory

• crow_eye_integration.py - Main integration bridge
• case_initializer.py - Initialize correlation engine for a case
• auto_feather_generator.py - Auto-generate feathers from Crow-Eye data
• feather_mappings.py - Define mappings for feather integration
• correlation_integration.py - Core correlation integration logic

Choose Identity-Based Engine When:

• ✅ Dataset has > 1,000 records
• ✅ Performance is critical
• ✅ You need identity tracking across artifacts
• ✅ You want to filter by specific applications
• ✅ Memory constraints require streaming mode
• ✅ Production-ready solution needed

Choose Time-Based Engine When: ✓ Production Ready

• ✅ Dataset has > 1,000 records
• ✅ Systematic temporal analysis across large datasets is needed
• ✅ Performance and memory efficiency are paramount
• ✅ Automated correlation pipelines are in use
• ✅ You require an O(N log N) scalable solution