From 08ad3b3b197eb78dfd32e7461f313d3017e6ae50 Mon Sep 17 00:00:00 2001 From: Cyril Achard Date: Tue, 10 Mar 2026 08:32:17 +0100 Subject: [PATCH 1/3] Remove outdated docs --- docs/README.md | 261 --------------------- docs/aravis_backend.md | 202 ----------------- docs/camera_support.md | 76 ------- docs/features.md | 473 --------------------------------------- docs/timestamp_format.md | 79 ------- docs/user_guide.md | 282 ----------------------- 6 files changed, 1373 deletions(-) delete mode 100644 docs/README.md delete mode 100644 docs/aravis_backend.md delete mode 100644 docs/camera_support.md delete mode 100644 docs/features.md delete mode 100644 docs/timestamp_format.md delete mode 100644 docs/user_guide.md diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index 8448701..0000000 --- a/docs/README.md +++ /dev/null @@ -1,261 +0,0 @@ -# DeepLabCut-live-GUI Documentation Index - -Welcome to the DeepLabCut-live-GUI documentation! This index will help you find the information you need. - -## Getting Started - -### New Users -1. **[README](../README.md)** - Project overview, installation, and quick start -2. **[User Guide](user_guide.md)** - Step-by-step walkthrough of all features - -### Quick References -- **[ARAVIS_QUICK_REF](../ARAVIS_QUICK_REF.md)** - Aravis backend quick reference -- **[Features Overview](features.md)** - Complete feature documentation - -## Core Documentation - -### Camera Setup -- **[Camera Support](camera_support.md)** - Overview of all camera backends -- **[Aravis Backend](aravis_backend.md)** - Linux/macOS GenICam camera setup -- Platform-specific guides for industrial cameras - -### Application Features -- **[Features Documentation](features.md)** - Detailed feature descriptions: - - Camera control and backends - - Real-time pose estimation - - Video recording - - Configuration management - - Processor system - - User interface - - Performance monitoring - - Advanced features - -### User Guide -- **[User Guide](user_guide.md)** - Complete usage walkthrough: - - Getting started - - Camera setup - - DLCLive configuration - - Recording videos - - Configuration management - - Common workflows - - Tips and best practices - - Troubleshooting - -## Advanced Topics - -### Processor System -- **[Processor Plugins](PLUGIN_SYSTEM.md)** - Custom pose processing -- **[Processor Auto-Recording](processor_auto_recording.md)** - Event-triggered recording -- Socket processor documentation - -### Technical Details -- **[Timestamp Format](timestamp_format.md)** - Synchronization and timing -- **[ARAVIS_BACKEND_SUMMARY](../ARAVIS_BACKEND_SUMMARY.md)** - Implementation details - -## By Use Case - -### I want to... - -#### Set up a camera -→ [Camera Support](camera_support.md) → Select backend → Follow setup guide - -**By Platform**: -- **Windows**: [README](../README.md#windows-gentl-for-industrial-cameras) → GenTL setup -- **Linux**: [Aravis Backend](aravis_backend.md) → Installation for Ubuntu/Debian -- **macOS**: [Aravis Backend](aravis_backend.md) → Installation via Homebrew - -**By Camera Type**: -- **Webcam**: [User Guide](user_guide.md#camera-setup) → OpenCV backend -- **Industrial Camera**: [Camera Support](camera_support.md) → GenTL/Aravis -- **Basler Camera**: [Camera Support](camera_support.md#basler-cameras) → pypylon setup -- **The Imaging Source**: [Aravis Backend](aravis_backend.md) or GenTL - -#### Run pose estimation -→ [User Guide](user_guide.md#dlclive-configuration) → Load model → Start inference - -#### Record high-speed video -→ [Features](features.md#video-recording) → Hardware encoding → GPU setup -→ [User Guide](user_guide.md#high-speed-recording-60-fps) → Optimization tips - -#### Create custom processor -→ [Processor Plugins](PLUGIN_SYSTEM.md) → Plugin architecture → Examples - -#### Trigger recording remotely -→ [Features](features.md#auto-recording-feature) → Auto-recording setup -→ Socket processor documentation - -#### Optimize performance -→ [Features](features.md#performance-optimization) → Metrics → Adjustments -→ [User Guide](user_guide.md#tips-and-best-practices) → Best practices - -## By Topic - -### Camera Backends -| Backend | Documentation | Platform | -|---------|---------------|----------| -| OpenCV | [User Guide](user_guide.md#step-1-select-camera-backend) | All | -| GenTL | [Camera Support](camera_support.md) | Windows, Linux | -| Aravis | [Aravis Backend](aravis_backend.md) | Linux, macOS | -| Basler | [Camera Support](camera_support.md#basler-cameras) | All | - -### Configuration -- **Basics**: [README](../README.md#configuration) -- **Management**: [User Guide](user_guide.md#working-with-configurations) -- **Templates**: [User Guide](user_guide.md#configuration-templates) -- **Details**: [Features](features.md#configuration-management) - -### Recording -- **Quick Start**: [User Guide](user_guide.md#recording-videos) -- **Features**: [Features](features.md#video-recording) -- **Optimization**: [README](../README.md#performance-optimization) -- **Auto-Recording**: [Features](features.md#auto-recording-feature) - -### DLCLive -- **Setup**: [User Guide](user_guide.md#dlclive-configuration) -- **Models**: [Features](features.md#model-support) -- **Performance**: [Features](features.md#performance-metrics) -- **Visualization**: [Features](features.md#pose-visualization) - -## Troubleshooting - -### Quick Fixes -1. **Camera not detected** → [User Guide](user_guide.md#troubleshooting-guide) -2. **Slow inference** → [Features](features.md#performance-optimization) -3. **Dropped frames** → [README](../README.md#troubleshooting) -4. **Recording issues** → [User Guide](user_guide.md#troubleshooting-guide) - -### Detailed Troubleshooting -- [User Guide - Troubleshooting Section](user_guide.md#troubleshooting-guide) -- [README - Troubleshooting](../README.md#troubleshooting) -- [Aravis Backend - Troubleshooting](aravis_backend.md#troubleshooting) - -## Development - -### Architecture -- **Project Structure**: [README](../README.md#development) -- **Backend Development**: [Camera Support](camera_support.md#contributing-new-camera-types) -- **Processor Development**: [Processor Plugins](PLUGIN_SYSTEM.md) - -### Implementation Details -- **Aravis Backend**: [ARAVIS_BACKEND_SUMMARY](../ARAVIS_BACKEND_SUMMARY.md) -- **Thread Safety**: [Features](features.md#thread-safety) -- **Resource Management**: [Features](features.md#resource-management) - -## Reference - -### Configuration Schema -```json -{ - "camera": { - "name": "string", - "index": "number", - "fps": "number", - "backend": "opencv|gentl|aravis|basler", - "exposure": "number (μs, 0=auto)", - "gain": "number (0.0=auto)", - "crop_x0/y0/x1/y1": "number", - "max_devices": "number", - "properties": "object" - }, - "dlc": { - "model_path": "string", - "model_type": "base|pytorch", - "additional_options": "object" - }, - "recording": { - "enabled": "boolean", - "directory": "string", - "filename": "string", - "container": "mp4|avi|mov", - "codec": "h264_nvenc|libx264|hevc_nvenc", - "crf": "number (0-51)" - }, - "bbox": { - "enabled": "boolean", - "x0/y0/x1/y1": "number" - } -} -``` - -### Performance Metrics -- **Camera FPS**: [Features](features.md#camera-metrics) -- **DLC Metrics**: [Features](features.md#dlc-metrics) -- **Recording Metrics**: [Features](features.md#recording-metrics) - -### Keyboard Shortcuts -| Action | Shortcut | -|--------|----------| -| Load configuration | Ctrl+O | -| Save configuration | Ctrl+S | -| Save as | Ctrl+Shift+S | -| Quit | Ctrl+Q | - -## External Resources - -### DeepLabCut -- [DeepLabCut](http://www.mackenziemathislab.org/deeplabcut) -- [DeepLabCut-live](https://github.com/DeepLabCut/DeepLabCut-live) -- [DeepLabCut Documentation](http://deeplabcut.github.io/DeepLabCut/docs/intro.html) - -### Camera Libraries -- [Aravis Project](https://github.com/AravisProject/aravis) -- [Harvesters (GenTL)](https://github.com/genicam/harvesters) -- [pypylon (Basler)](https://github.com/basler/pypylon) -- [OpenCV](https://opencv.org/) - -### Video Encoding -- [FFmpeg](https://ffmpeg.org/) -- [NVENC (NVIDIA)](https://developer.nvidia.com/nvidia-video-codec-sdk) - -## Getting Help - -### Support Channels -1. Check relevant documentation (use this index!) -2. Search GitHub issues -3. Review example configurations -4. Contact maintainers - -### Reporting Issues -When reporting bugs, include: -- GUI version -- Platform (OS, Python version) -- Camera backend and model -- Configuration file (if applicable) -- Error messages -- Steps to reproduce - -## Contributing - -Interested in contributing? -- See [README - Contributing](../README.md#contributing) -- Review [Development Section](../README.md#development) -- Check open GitHub issues -- Read coding guidelines - ---- - -## Document Version History - -- **v1.0** - Initial comprehensive documentation - - Complete README overhaul - - User guide creation - - Features documentation - - Camera backend guides - - Aravis backend implementation - -## Quick Navigation - -**Popular Pages**: -- [User Guide](user_guide.md) - Most comprehensive walkthrough -- [Features](features.md) - All capabilities detailed -- [Aravis Setup](aravis_backend.md) - Linux industrial cameras -- [Camera Support](camera_support.md) - All camera backends - -**By Experience Level**: -- **Beginner**: [README](../README.md) → [User Guide](user_guide.md) -- **Intermediate**: [Features](features.md) → [Camera Support](camera_support.md) -- **Advanced**: [Processor Plugins](PLUGIN_SYSTEM.md) → Implementation details - ---- - -*Last updated: 2025-10-24* diff --git a/docs/aravis_backend.md b/docs/aravis_backend.md deleted file mode 100644 index 67024ba..0000000 --- a/docs/aravis_backend.md +++ /dev/null @@ -1,202 +0,0 @@ -# Aravis Backend - -The Aravis backend provides support for GenICam-compatible cameras using the [Aravis](https://github.com/AravisProject/aravis) library. - -## Features - -- Support for GenICam/GigE Vision cameras -- Automatic device detection with `get_device_count()` -- Configurable exposure time and gain -- Support for various pixel formats (Mono8, Mono12, Mono16, RGB8, BGR8) -- Efficient streaming with configurable buffer count -- Timeout handling for robust operation - -## Installation - -### Linux (Ubuntu/Debian) -```bash -sudo apt-get install gir1.2-aravis-0.8 python3-gi -``` - -### Linux (Fedora) -```bash -sudo dnf install aravis python3-gobject -``` - -### Windows -Aravis support on Windows requires building from source or using WSL. For native Windows support, consider using the GenTL backend instead. - -### macOS -```bash -brew install aravis -pip install pygobject -``` - -## Configuration - -### Basic Configuration - -Select "aravis" as the backend in the GUI or in your configuration file: - -```json -{ - "camera": { - "backend": "aravis", - "index": 0, - "fps": 30.0, - "exposure": 10000, - "gain": 5.0 - } -} -``` - -### Advanced Properties - -You can configure additional Aravis-specific properties via the `properties` dictionary: - -```json -{ - "camera": { - "backend": "aravis", - "index": 0, - "fps": 30.0, - "exposure": 10000, - "gain": 5.0, - "properties": { - "camera_id": "MyCamera-12345", - "pixel_format": "Mono8", - "timeout": 2000000, - "n_buffers": 10 - } - } -} -``` - -#### Available Properties - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `camera_id` | string | None | Specific camera ID to open (overrides index) | -| `pixel_format` | string | "Mono8" | Pixel format: Mono8, Mono12, Mono16, RGB8, BGR8 | -| `timeout` | int | 2000000 | Frame timeout in microseconds (2 seconds) | -| `n_buffers` | int | 10 | Number of buffers in the acquisition stream | - -### Exposure and Gain - -The Aravis backend supports exposure time (in microseconds) and gain control: - -- **Exposure**: Set via the GUI exposure field or `settings.exposure` (0 = auto, >0 = manual in μs) -- **Gain**: Set via the GUI gain field or `settings.gain` (0.0 = auto, >0.0 = manual value) - -When exposure or gain are set to non-zero values, the backend automatically disables auto-exposure and auto-gain. - -## Camera Selection - -### By Index -The default method is to select cameras by index (0, 1, 2, etc.): -```json -{ - "camera": { - "backend": "aravis", - "index": 0 - } -} -``` - -### By Camera ID -You can also select a specific camera by its ID: -```json -{ - "camera": { - "backend": "aravis", - "properties": { - "camera_id": "TheImagingSource-12345678" - } - } -} -``` - -## Supported Pixel Formats - -The backend automatically converts different pixel formats to BGR format for consistency: - -- **Mono8**: 8-bit grayscale → BGR -- **Mono12**: 12-bit grayscale → scaled to 8-bit → BGR -- **Mono16**: 16-bit grayscale → scaled to 8-bit → BGR -- **RGB8**: 8-bit RGB → BGR (color conversion) -- **BGR8**: 8-bit BGR (no conversion needed) - -## Performance Tuning - -### Buffer Count -Increase `n_buffers` for high-speed cameras or systems with variable latency: -```json -{ - "properties": { - "n_buffers": 20 - } -} -``` - -### Timeout -Adjust timeout for slower cameras or network cameras: -```json -{ - "properties": { - "timeout": 5000000 - } -} -``` -(5 seconds = 5,000,000 microseconds) - -## Troubleshooting - -### No cameras detected -1. Verify Aravis installation: `arv-tool-0.8 -l` -2. Check camera is powered and connected -3. Ensure proper network configuration for GigE cameras -4. Check user permissions for USB cameras - -### Timeout errors -- Increase the `timeout` property -- Check network bandwidth for GigE cameras -- Verify camera is properly configured and streaming - -### Pixel format errors -- Check camera's supported pixel formats: `arv-tool-0.8 -n features` -- Try alternative formats: Mono8, RGB8, etc. - -## Comparison with GenTL Backend - -| Feature | Aravis | GenTL | -|---------|--------|-------| -| Platform | Linux (best), macOS | Windows (best), Linux | -| Camera Support | GenICam/GigE | GenTL producers | -| Installation | System packages | Vendor CTI files | -| Performance | Excellent | Excellent | -| Auto-detection | Yes | Yes | - -## Example: The Imaging Source Camera - -```json -{ - "camera": { - "backend": "aravis", - "index": 0, - "fps": 60.0, - "exposure": 8000, - "gain": 10.0, - "properties": { - "pixel_format": "Mono8", - "n_buffers": 15, - "timeout": 3000000 - } - } -} -``` - -## Resources - -- [Aravis Project](https://github.com/AravisProject/aravis) -- [GenICam Standard](https://www.emva.org/standards-technology/genicam/) -- [Python GObject Documentation](https://pygobject.readthedocs.io/) diff --git a/docs/camera_support.md b/docs/camera_support.md deleted file mode 100644 index ed0b1e0..0000000 --- a/docs/camera_support.md +++ /dev/null @@ -1,76 +0,0 @@ -## Camera Support - -DeepLabCut-live-GUI supports multiple camera backends for different platforms and camera types: - -### Supported Backends - -1. **OpenCV** - Universal webcam and USB camera support (all platforms) -2. **GenTL** - Industrial cameras via GenTL producers (Windows, Linux) -3. **Aravis** - GenICam/GigE Vision cameras (Linux, macOS) -4. **Basler** - Basler cameras via pypylon (all platforms) - -### Backend Selection - -You can select the backend in the GUI from the "Backend" dropdown, or in your configuration file: - -```json -{ - "camera": { - "backend": "aravis", - "index": 0, - "fps": 30.0 - } -} -``` - -### Platform-Specific Recommendations - -#### Windows -- **OpenCV compatible cameras**: Best for webcams and simple USB cameras. OpenCV is installed with DeepLabCut-live-GUI. -- **GenTL backend**: Recommended for industrial cameras (The Imaging Source, Basler, etc.) via vendor-provided CTI files. -- **Basler cameras**: Can use either GenTL or pypylon backend. - -#### Linux -- **OpenCV compatible cameras**: Good for webcams via Video4Linux drivers. Installed with DeepLabCut-live-GUI. -- **Aravis backend**: **Recommended** for GenICam/GigE Vision industrial cameras (The Imaging Source, Basler, Point Grey, etc.) - - Easy installation via system package manager - - Better Linux support than GenTL - - See [Aravis Backend Documentation](aravis_backend.md) -- **GenTL backend**: Alternative for industrial cameras if vendor provides Linux CTI files. - -#### macOS -- **OpenCV compatible cameras**: For webcams and compatible USB cameras. -- **Aravis backend**: For GenICam/GigE Vision cameras (requires Homebrew installation). - -### Quick Installation Guide - -#### Aravis (Linux/Ubuntu) -```bash -sudo apt-get install gir1.2-aravis-0.8 python3-gi -``` - -#### Aravis (macOS) -```bash -brew install aravis -pip install pygobject -``` - -#### GenTL (Windows) -Install vendor-provided camera drivers and SDK. CTI files are typically in: -- `C:\Program Files\The Imaging Source Europe GmbH\IC4 GenTL Driver\bin\` - -### Backend Comparison - -| Feature | OpenCV | GenTL | Aravis | Basler (pypylon) | -|---------|--------|-------|--------|------------------| -| Exposure control | No | Yes | Yes | Yes | -| Gain control | No | Yes | Yes | Yes | -| Windows | ✅ | ✅ | ❌ | ✅ | -| Linux | ✅ | ✅ | ✅ | ✅ | -| macOS | ✅ | ❌ | ✅ | ✅ | - -### Detailed Backend Documentation - -- [Aravis Backend](aravis_backend.md) - GenICam/GigE cameras on Linux/macOS -- GenTL Backend - Industrial cameras via vendor CTI files -- OpenCV Backend - Universal webcam support diff --git a/docs/features.md b/docs/features.md deleted file mode 100644 index 91d87ad..0000000 --- a/docs/features.md +++ /dev/null @@ -1,473 +0,0 @@ -# DeepLabCut-live-GUI Features - -## Table of Contents - -- [Camera Control](#camera-control) -- [Real-Time Pose Estimation](#real-time-pose-estimation) -- [Video Recording](#video-recording) -- [Configuration Management](#configuration-management) -- [Processor System](#processor-system) -- [User Interface](#user-interface) -- [Performance Monitoring](#performance-monitoring) -- [Advanced Features](#advanced-features) - ---- - -## Camera Control - -### Multi-Backend Support - -The GUI supports four different camera backends, each optimized for different use cases: - -#### OpenCV Backend -- **Platform**: Windows, Linux -- **Best For**: Webcams, simple USB cameras -- **Installation**: Built-in with OpenCV -- **Limitations**: Limited exposure/gain control - -#### GenTL Backend (Harvesters) -- **Platform**: Windows, Linux -- **Best For**: Industrial cameras with GenTL producers -- **Installation**: Requires vendor CTI files -- **Features**: Full camera control, smart device detection - -#### Aravis Backend -- **Platform**: Linux (best) -- **Best For**: GenICam/GigE Vision cameras -- **Installation**: System packages (`gir1.2-aravis-0.8`) - -#### Basler Backend (pypylon) -- **Platform**: Windows, Linux, macOS -- **Best For**: Basler cameras specifically -- **Installation**: Pylon SDK + pypylon -- **Features**: Vendor-specific optimizations - -### Camera Settings - -#### Frame Rate Control -- Range: 1-240 FPS (hardware dependent) -- Real-time FPS monitoring -- Automatic camera validation - -#### Exposure Control -- Auto mode (value = 0) -- Manual mode (microseconds) -- Range: 0-1,000,000 μs -- Real-time adjustment (backend dependent) - -#### Gain Control -- Auto mode (value = 0.0) -- Manual mode (gain value) -- Range: 0.0-100.0 -- Useful for low-light conditions - -#### Region of Interest (ROI) Cropping -- Define crop region: (x0, y0, x1, y1) -- Applied before recording and inference -- Reduces processing load -- Maintains aspect ratio - -#### Image Rotation -- 0°, 90°, 180°, 270° rotation -- Applied to all outputs -- Useful for mounted cameras - -### Smart Camera Detection - -The GUI intelligently detects available cameras: - -1. **Backend-Specific**: Each backend reports available cameras -2. **No Blind Probing**: GenTL and Aravis query actual device count -3. **Fast Refresh**: Only check connected devices -4. **Detailed Labels**: Shows vendor, model, serial number - -Example detection output: -``` -[CameraDetection] Available cameras for backend 'gentl': - ['0:DMK 37BUX287 (26320523)', '1:Basler acA1920 (40123456)'] -``` - ---- - -## Real-Time Pose Estimation - -### DLCLive Integration - -#### Model Support -- **PyTorch**: PyTorch-exported models -- Model selection via dropdown -- Automatic model validation - -#### Inference Pipeline -1. **Frame Acquisition**: Camera thread → Queue -2. **Preprocessing**: Crop, resize (optional) -3. **Inference**: DLCLive model processing -4. **Pose Output**: (x, y) coordinates per keypoint -5. **Visualization**: Optional overlay on video - -#### Performance Metrics -- **Inference FPS**: Actual processing rate -- **Latency**: Time from capture to pose output - - Last latency (ms) - - Average latency (ms) -- **Queue Status**: Frame buffer depth -- **Dropped Frames**: Count of skipped frames - -### Pose Visualization - -#### Overlay Options -- **Toggle**: "Display pose predictions" checkbox -- **Keypoint Markers**: Green circles at (x, y) positions -- **Real-Time Update**: Synchronized with video feed -- **No Performance Impact**: Rendering optimized - -#### Bounding Box Visualization -- **Purpose**: Visual ROI definition -- **Configuration**: (x0, y0, x1, y1) coordinates -- **Color**: Red rectangle overlay -- **Use Cases**: - - Crop region preview - - Analysis area marking - - Multi-region tracking - -### Initialization Feedback - -Visual indicators during model loading: -1. **"Initializing DLCLive!"** - Blue button during load -2. **"DLCLive running!"** - Green button when ready -3. Status bar updates with progress - ---- - -## Video Recording - -### Recording Capabilities - -#### Hardware-Accelerated Encoding -- **NVENC (NVIDIA)**: GPU-accelerated H.264/H.265 - - Codecs: `h264_nvenc`, `hevc_nvenc` - - 10x faster than software encoding - - Minimal CPU usage -- **Software Encoding**: CPU-based fallback - - Codecs: `libx264`, `libx265` - - Universal compatibility - -#### Container Formats -- **MP4**: Most compatible, web-ready -- **AVI**: Legacy support -- **MOV**: Apple ecosystem - -#### Quality Control -- **CRF (Constant Rate Factor)**: 0-51 - - 0 = Lossless (huge files) - - 23 = Default (good quality) - - 28 = High compression - - 51 = Lowest quality -- **Presets**: ultrafast, fast, medium, slow - -### Recording Features - -#### Timestamp Synchronization -- Frame-accurate timestamps -- Microsecond precision -- Synchronized with pose data -- Stored in separate files - -#### Performance Monitoring -- **Write FPS**: Actual encoding rate -- **Queue Size**: Buffer depth (~ms) -- **Latency**: Encoding delay -- **Frames Written/Enqueued**: Progress tracking -- **Dropped Frames**: Quality indicator - -#### Buffer Management -- Configurable queue size -- Automatic overflow handling -- Warning on frame drops -- Backpressure indication - -### Auto-Recording Feature - -Processor-triggered recording: - -1. **Enable**: Check "Auto-record video on processor command" -2. **Processor Control**: Custom processor sets recording flag -3. **Automatic Start**: GUI starts recording when flag set -4. **Session Naming**: Uses processor-defined session name -5. **Automatic Stop**: GUI stops when flag cleared - -**Use Cases**: -- Event-triggered recording -- Trial-based experiments -- Conditional data capture -- Remote control via socket - ---- - -## Configuration Management - -### Configuration File Structure - -Single JSON file contains all settings: - -```json -{ - "camera": { ... }, - "dlc": { ... }, - "recording": { ... }, - "bbox": { ... } -} -``` - -### Features - -#### Save/Load Operations -- **Load**: File → Load configuration (Ctrl+O) -- **Save**: File → Save configuration (Ctrl+S) -- **Save As**: File → Save configuration as (Ctrl+Shift+S) -- **Auto-sync**: GUI fields update from file - -#### Multiple Configurations -- Switch between experiments quickly -- Per-animal configurations -- Environment-specific settings -- Backup and version control - -#### Validation -- Type checking on load -- Default values for missing fields -- Error messages for invalid entries -- Safe fallback to defaults ---- - -## Processor System - -### Plugin Architecture - -Custom pose processors for real-time analysis and control. - -#### Processor Interface - -```python -class MyProcessor: - """Custom processor example.""" - - def process(self, pose, timestamp): - """Process pose data in real-time. - - Args: - pose: numpy array (n_keypoints, 3) - x, y, likelihood - timestamp: float - frame timestamp - """ - # Extract keypoint positions - nose_x, nose_y = pose[0, :2] - - # Custom logic - if nose_x > 320: - self.trigger_event() - - # Return results (optional) - return {"position": (nose_x, nose_y)} -``` - -#### Loading Processors - -1. Place processor file in `dlclivegui/processors/` -2. Click "Refresh" in processor dropdown -3. Select processor from list -4. Start inference to activate - -#### Built-in Processors - -**Socket Processor** (`dlc_processor_socket.py`): -- TCP socket server for remote control -- Commands: `START_RECORDING`, `STOP_RECORDING` -- Session management -- Multi-client support - -### Auto-Recording Integration - -Processors can control recording: - -```python -class RecordingProcessor: - def __init__(self): - self._vid_recording = False - self.session_name = "default" - - @property - def video_recording(self): - return self._vid_recording - - def start_recording(self, session): - self.session_name = session - self._vid_recording = True - - def stop_recording(self): - self._vid_recording = False -``` - -The GUI monitors `video_recording` property and automatically starts/stops recording. - ---- - -## User Interface - -### Layout - -#### Control Panel (Left) -- **Camera Settings**: Backend, index, FPS, exposure, gain, crop -- **DLC Settings**: Model path, type, processor, options -- **Recording Settings**: Path, filename, codec, quality -- **Bounding Box**: Visualization controls - -#### Video Display (Right) -- Live camera feed -- Pose overlay (optional) -- Bounding box overlay (optional) -- Auto-scaling to window size - -#### Status Bar (Bottom) -- Current operation status -- Error messages -- Success confirmations - -### Control Groups - -#### Camera Controls -- Backend selection dropdown -- Camera index/refresh -- FPS, exposure, gain spinboxes -- Crop coordinates -- Rotation selector -- **Start/Stop Preview** buttons - -#### DLC Controls -- Model path browser -- Model type selector -- Processor folder/selection -- Additional options (JSON) -- **Start/Stop Inference** buttons -- "Display pose predictions" checkbox -- "Auto-record" checkbox -- Processor status display - -#### Recording Controls -- Output directory browser -- Filename input -- Container/codec selectors -- CRF quality slider -- **Start/Stop Recording** buttons - -### Visual Feedback - -#### Button States -- **Disabled**: Gray, not clickable -- **Enabled**: Default color, clickable -- **Active**: - - Preview running: Stop button enabled - - Inference initializing: Blue "Initializing DLCLive!" - - Inference ready: Green "DLCLive running!" - -#### Status Indicators -- Camera FPS (last 5 seconds) -- DLC performance metrics -- Recording statistics -- Processor connection status - ---- - -## Performance Monitoring - -### Real-Time Metrics - -#### Camera Metrics -- **Throughput**: FPS over last 5 seconds -- **Formula**: `(frame_count - 1) / time_elapsed` -- **Display**: "45.2 fps (last 5 s)" - -#### DLC Metrics -- **Inference FPS**: Poses processed per second -- **Latency**: - - Last frame latency (ms) - - Average latency over session (ms) -- **Queue**: Number of frames waiting -- **Dropped**: Frames skipped due to queue full -- **Format**: "150/152 frames | inference 42.1 fps | latency 23.5 ms (avg 24.1 ms) | queue 2 | dropped 2" - -#### Recording Metrics -- **Write FPS**: Encoding rate -- **Frames**: Written/Enqueued ratio -- **Latency**: Encoding delay (ms) -- **Buffer**: Queue size (~milliseconds) -- **Dropped**: Encoding failures -- **Format**: "1500/1502 frames | write 59.8 fps | latency 12.3 ms (avg 12.5 ms) | queue 5 (~83 ms) | dropped 2" - ---- - -## Advanced Features - -### Frame Synchronization - -All components share frame timestamps: -- Camera controller generates timestamps -- DLC processor preserves timestamps -- Video recorder stores timestamps -- Enables post-hoc alignment - -### Error Recovery - -#### Camera Connection Loss -- Automatic detection via frame grab failure -- User notification -- Clean resource cleanup -- Restart capability - -#### Recording Errors -- Frame size mismatch detection -- Automatic recovery with new settings -- Warning display -- No data loss - -### Thread Safety - -Multi-threaded architecture: -- **Main Thread**: GUI event loop -- **Camera Thread**: Frame acquisition -- **DLC Thread**: Pose inference -- **Recording Thread**: Video encoding - -Qt signals/slots ensure thread-safe communication. - -### Resource Management - -#### Automatic Cleanup -- Camera release on stop/error -- DLC model unload on stop -- Recording finalization -- Thread termination - -#### Memory Management -- Bounded queues prevent memory leaks -- Frame copy-on-write -- Efficient numpy array handling - -### Extensibility - -### Debugging Features - -#### Logging -- Console output for errors -- Frame acquisition logging -- Performance warnings -- Connection status ---- ---- - -## Keyboard Shortcuts - -- **Ctrl+O**: Load configuration -- **Ctrl+S**: Save configuration -- **Ctrl+Shift+S**: Save configuration as -- **Ctrl+Q**: Quit application ---- diff --git a/docs/timestamp_format.md b/docs/timestamp_format.md deleted file mode 100644 index ac5e5a7..0000000 --- a/docs/timestamp_format.md +++ /dev/null @@ -1,79 +0,0 @@ -# Video Frame Timestamp Format - -When recording video, the application automatically saves frame timestamps to a JSON file alongside the video file. - -## File Naming - -For a video file named `recording_2025-10-23_143052.mp4`, the timestamp file will be: -``` -recording_2025-10-23_143052.mp4_timestamps.json -``` - -## JSON Structure - -```json -{ - "video_file": "recording_2025-10-23_143052.mp4", - "num_frames": 1500, - "timestamps": [ - 1729693852.123456, - 1729693852.156789, - 1729693852.190123, - ... - ], - "start_time": 1729693852.123456, - "end_time": 1729693902.123456, - "duration_seconds": 50.0 -} -``` - -## Fields - -- **video_file**: Name of the associated video file -- **num_frames**: Total number of frames recorded -- **timestamps**: Array of Unix timestamps (seconds since epoch with microsecond precision) for each frame -- **start_time**: Timestamp of the first frame -- **end_time**: Timestamp of the last frame -- **duration_seconds**: Total recording duration in seconds - -## Usage - -The timestamps correspond to the exact time each frame was captured by the camera (from `FrameData.timestamp`). This allows precise synchronization with: - -- DLC pose estimation results -- External sensors or triggers -- Other data streams recorded during the same session - -## Example: Loading Timestamps in Python - -```python -import json -from datetime import datetime - -# Load timestamps -with open('recording_2025-10-23_143052.mp4_timestamps.json', 'r') as f: - data = json.load(f) - -print(f"Video: {data['video_file']}") -print(f"Total frames: {data['num_frames']}") -print(f"Duration: {data['duration_seconds']:.2f} seconds") - -# Convert first timestamp to human-readable format -start_dt = datetime.fromtimestamp(data['start_time']) -print(f"Recording started: {start_dt.isoformat()}") - -# Calculate average frame rate -avg_fps = data['num_frames'] / data['duration_seconds'] -print(f"Average FPS: {avg_fps:.2f}") - -# Access individual frame timestamps -for frame_idx, timestamp in enumerate(data['timestamps']): - print(f"Frame {frame_idx}: {timestamp}") -``` - -## Notes - -- Timestamps use `time.time()` which returns Unix epoch time with high precision -- Frame timestamps are captured when frames arrive from the camera, before any processing -- If frames are dropped due to queue overflow, those frames will not have timestamps in the array -- The timestamp array length should match the number of frames in the video file diff --git a/docs/user_guide.md b/docs/user_guide.md deleted file mode 100644 index b6f1905..0000000 --- a/docs/user_guide.md +++ /dev/null @@ -1,282 +0,0 @@ -# DeepLabCut-live-GUI User Guide - -Complete walkthrough for using the DeepLabCut-live-GUI application. - -## Table of Contents - -1. [Getting Started](#getting-started) -2. [Camera Setup](#camera-setup) -3. [DLCLive Configuration](#dlclive-configuration) -4. [Recording Videos](#recording-videos) ---- - -## Getting Started - -### First Launch - -1. Open a terminal/command prompt -2. Run the application: - ```bash - dlclivegui - ``` -3. The main window will appear with three control panels and a video display area - -### Interface Overview - -``` -┌─────────────────────────────────────────────────────┐ -│ File Help │ -├─────────────┬───────────────────────────────────────┤ -│ Camera │ │ -│ Settings │ │ -│ │ │ -│ ─────────── │ Video Display │ -│ DLCLive │ │ -│ Settings │ │ -│ │ │ -│ ─────────── │ │ -│ Recording │ │ -│ Settings │ │ -│ │ │ -│ ─────────── │ │ -│ Bounding │ │ -│ Box │ │ -│ │ │ -│ ─────────── │ │ -│ [Preview] │ │ -│ [Stop] │ │ -└─────────────┴───────────────────────────────────────┘ -│ Status: Ready │ -└─────────────────────────────────────────────────────┘ -``` - ---- - -## Camera Setup - -### Step 1: Select Camera Backend - -The **Backend** dropdown shows available camera drivers: - -| Backend | When to Use | -|---------|-------------| -| **opencv** | Webcams, USB cameras (universal) | -| **gentl** | Industrial cameras (Windows/Linux) | -| **aravis** | GenICam/GigE cameras (Linux/macOS) | -| **basler** | Basler cameras specifically | - -**Note**: Unavailable backends appear grayed out. Install required drivers to enable them. - -### Step 2: Select Camera - -1. Click **Refresh** next to the camera dropdown -2. Wait for camera detection (1-3 seconds) -3. Select your camera from the dropdown - -The list shows camera details: -``` -0:DMK 37BUX287 (26320523) -│ │ └─ Serial Number -│ └─ Model Name -└─ Index -``` - -### Step 3: Configure Camera Parameters - -#### Frame Rate -- **Range**: 1-240 FPS (hardware dependent) -- **Recommendation**: Start with 30 FPS, increase as needed -- **Note**: Higher FPS = more processing load - -#### Exposure Time -- **Auto**: Set to 0 (default) -- **Manual**: Microseconds (e.g., 10000 = 10ms) -- **Tips**: - - Shorter exposure = less motion blur - - Longer exposure = better low-light performance - - Typical range: 5,000-30,000 μs - -#### Gain -- **Auto**: Set to 0.0 (default) -- **Manual**: 0.0-100.0 -- **Tips**: - - Higher gain = brighter image but more noise - - Start low (5-10) and increase if needed - - Auto mode works well for most cases - -#### Cropping (Optional) -Reduce frame size for faster processing: - -1. Set crop region: (x0, y0, x1, y1) - - x0, y0: Top-left corner - - x1, y1: Bottom-right corner -2. Use Bounding Box visualization to preview -3. Set all to 0 to disable cropping - -**Example**: Crop to center 640x480 region of 1280x720 camera: -``` -x0: 320 -y0: 120 -x1: 960 -y1: 600 -``` - -#### Rotation -Select if camera is mounted at an angle: -- 0° (default) -- 90° (rotated right) -- 180° (upside down) -- 270° (rotated left) - -### Step 4: Start Camera Preview - -1. Click **Start Preview** -2. Video feed should appear in the display area -3. Check the **Throughput** metric below camera settings -4. Verify frame rate matches expected value - -**Troubleshooting**: -- **No preview**: Check camera connection and permissions -- **Low FPS**: Reduce resolution or increase exposure time -- **Black screen**: Check exposure settings -- **Distorted image**: Verify backend compatibility - ---- - -## DLCLive Configuration - -### Prerequisites - -1. Exported DLCLive model (see [DLC documentation](https://github.com/DeepLabCut/DeepLabCut/blob/main/docs/HelperFunctions.md#model-export-function)) -2. DeepLabCut-live installed (`pip install deeplabcut-live`) -3. Camera preview running - -### Step 1: Select Model - -1. Click **Browse** next to "Model directory" -2. Navigate to your exported DLCLive model folder -3. Select the folder containing: - - `pose_cfg.yaml` - - Model weights (`.pb`, `.pth`, etc.) - -### Step 2: Choose Model Type -We only support newer, pytorch based models. -- **PyTorch**: PyTorch-based models (requires PyTorch) - - -**Common options**: -- `processor`: "cpu" or "gpu" -- `resize`: Scale factor (0.5 = half size) -- `pcutoff`: Likelihood threshold -- `cropping`: Crop before inference - -### Step 4: Select Processor (Optional) - -If using custom pose processors: - -1. Click **Browse** next to "Processor folder" (or use default) -2. Click **Refresh** to scan for processors -3. Select processor from dropdown -4. Processor will activate when inference starts - -### Step 5: Start Inference - -1. Ensure camera preview is running -2. Click **Start pose inference** -3. Button changes to "Initializing DLCLive!" (blue) -4. Wait for model loading (5-30 seconds) -5. Button changes to "DLCLive running!" (green) -6. Check **Performance** metrics - -**Performance Metrics**: -``` -150/152 frames | inference 42.1 fps | latency 23.5 ms (avg 24.1 ms) | queue 2 | dropped 2 -``` -- **150/152**: Processed/Total frames -- **inference 42.1 fps**: Processing rate -- **latency 23.5 ms**: Current processing delay -- **queue 2**: Frames waiting -- **dropped 2**: Skipped frames (due to full queue) - -### Step 6: Enable Visualization (Optional) - -Check **"Display pose predictions"** to overlay keypoints on video. - -- Keypoints appear as green circles -- Updates in real-time with video -- Can be toggled during inference - ---- - -## Recording Videos - -### Basic Recording - -1. **Configure output path**: - - Click **Browse** next to "Output directory" - - Select or create destination folder - -2. **Set filename**: - - Enter base filename (e.g., "session_001") - - Extension added automatically based on container - -3. **Select format**: - - **Container**: mp4 (recommended), avi, mov - - **Codec**: - - `h264_nvenc` (NVIDIA GPU - fastest) - - `libx264` (CPU - universal) - - `hevc_nvenc` (NVIDIA H.265) - -4. **Set quality** (CRF slider): - - 0-17: Very high quality, large files - - 18-23: High quality (recommended) - - 24-28: Medium quality, smaller files - - 29-51: Lower quality, smallest files - -5. **Start recording**: - - Ensure camera preview is running - - Click **Start recording** - - **Stop recording** button becomes enabled - -6. **Monitor performance**: - - Check "Performance" metrics - - Watch for dropped frames - - Verify write FPS matches camera FPS - -### Advanced Recording Options - -#### High-Speed Recording (60+ FPS) - -**Settings**: -- Codec: `h264_nvenc` (requires NVIDIA GPU) -- CRF: 28 (higher compression) -- Crop region: Reduce frame size -- Close other applications - -#### High-Quality Recording - -**Settings**: -- Codec: `libx264` or `h264_nvenc` -- CRF: 18-20 -- Full resolution -- Sufficient disk space - - -### Auto-Recording - -Enable automatic recording triggered by processor events: - -1. **Select a processor** that supports auto-recording -2. **Enable**: Check "Auto-record video on processor command" -3. **Start inference**: Processor will control recording -4. **Session management**: Files named by processor - ---- -## Next Steps - -- Explore [Features Documentation](features.md) for detailed capabilities -- Review [Camera Backend Guide](camera_support.md) for advanced setup -- Check [Processor System](PLUGIN_SYSTEM.md) for custom processing -- See [Aravis Backend](aravis_backend.md) for Linux industrial cameras - ---- From 8848c0ec6b424bf8f1e0391cf7cf4308e7484d68 Mon Sep 17 00:00:00 2001 From: Cyril Achard Date: Tue, 10 Mar 2026 08:32:47 +0100 Subject: [PATCH 2/3] Add link to docs website --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 90bfcc3..03eea37 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ The new interface supports **multi-camera preview** with a **tiled display**, Py ## Documentation -> Full documentation is a work in progress and will be posted to DeepLabCut's official docs site when ready. +> Find the full documentation at the [DeepLabCut docs website](https://deeplabcut.github.io/DeepLabCut/docs/dlc-live/dlc-live-gui/index.html) --- From 759e7069c2f35c103d3285c31cc04fac6c73f6ef Mon Sep 17 00:00:00 2001 From: Cyril Achard Date: Tue, 10 Mar 2026 08:37:34 +0100 Subject: [PATCH 3/3] Remove separators in README --- README.md | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/README.md b/README.md index 03eea37..f7d5d3b 100644 --- a/README.md +++ b/README.md @@ -20,14 +20,10 @@ The GUI has been modernized and is now built with **PySide6 (Qt)** (replacing tk The new interface supports **multi-camera preview** with a **tiled display**, PyTorch models, and features improved interactive workflows for experimental use. ---- - ## Documentation > Find the full documentation at the [DeepLabCut docs website](https://deeplabcut.github.io/DeepLabCut/docs/dlc-live/dlc-live-gui/index.html) ---- - ## System requirements (quick summary) - **Python 3.10, 3.11 or 3.12** @@ -36,8 +32,6 @@ The new interface supports **multi-camera preview** with a **tiled display**, Py - **TensorFlow** *(for backwards compatibility with existing models; Windows installs are no longer available for Python > 3.10)* - A supported camera backend (OpenCV webcams by default; additional backends supported) ---- - ## Installation While the previous `deeplabcut-live-gui` is available on PyPI, the newest PySide6-based GUI and features **must be obtained by installing from source**. @@ -109,8 +103,6 @@ pip install -e .[pytorch] pip install -e .[tf] ``` ---- - ## Run the application > [!TIP] @@ -128,8 +120,6 @@ uv run dlclivegui > [!IMPORTANT] > Activate your venv/conda environment before launching so the GUI can access installed dependencies. ---- - ## Typical workflow The new GUI supports **one or more cameras**. @@ -152,8 +142,6 @@ Typical workflow: > For additional camera ecosystems (Basler, GenTL, Aravis, etc.), see the relevant documentation. ---- - ## Current limitations - Pose inference runs on **one selected camera at a time** (even in multi-camera mode) @@ -177,4 +165,4 @@ If you use this code, we kindly ask you to please cite: This project is under active development — feedback from real experimental use is highly valued. -Please report issues, suggest features, or contribute on [GitHub](https://github.com/DeepLabCut/DeepLabCut-live-GUI). +Please report issues, suggest features, or contribute here on [GitHub](https://github.com/DeepLabCut/DeepLabCut-live-GUI/issues).