Core Components

Core Components

OstrichDB is built with a modular architecture where each component has specific responsibilities. This document details each core component and how they work together.

Engine (src/core/engine/)

The engine is the heart of OstrichDB, containing the core database logic and data processing functionality.

Responsibilities

• Data processing: Core algorithms for data manipulation
• Query execution: Processing search and filter operations
• Transaction management: Ensuring data consistency
• Memory management: Efficient allocation and cleanup

Key Files

• Database initialization and startup
• Core data operation handlers
• Memory pool management
• Query processing engine

Server (src/core/server/)

The HTTP server component provides the REST API interface and handles all client communications.

Responsibilities

• HTTP request handling: Processing incoming API requests
• Response formatting: Converting internal data to JSON responses
• Routing: Mapping URLs to appropriate handlers
• Middleware: CORS, authentication, and logging
• Error handling: Converting internal errors to HTTP responses

Features

• RESTful API with complete CRUD operations
• CORS support for web applications
• Request/response logging
• Health check endpoints
• Graceful shutdown handling

Architecture

Client Request → Router → Authentication → Handler → Engine → Response

Data Layer (src/core/engine/data/)

The data layer manages the hierarchical data structures and file operations for all data types.

Components

Collections Manager

• Collection lifecycle: Create, read, update, delete operations
• Metadata management: Collection properties and configuration
• Encryption integration: Seamless encrypt/decrypt operations

Clusters Manager

• Cluster organization: Grouping and managing record clusters
• Query optimization: Efficient cluster-level searches
• Storage management: File system integration

Records Manager

• Record operations: CRUD operations for individual records
• Type validation: Ensuring data type integrity
• Indexing: Fast record lookup and retrieval

Data Storage

data/
├── projects/
│   └── {project_name}/
│       └── collections/
│           └── {collection_name}/
│               ├── metadata.json
│               └── clusters/
│                   └── {cluster_name}/
│                       ├── metadata.json
│                       └── records/
│                           └── {record_files}

Security (src/core/engine/security/)

The security component handles all cryptographic operations, authentication, and access control.

Encryption System

• Master key management: User-specific encryption keys
• AES encryption: Industry-standard encryption algorithms
• Key derivation: Secure key generation from user credentials
• Automatic operations: Transparent encrypt/decrypt cycles

Authentication

• JWT tokens: Stateless authentication system
• Token validation: Request-level authentication
• User management: User registration and login
• Session handling: Token refresh and expiration

Access Control

• Project ownership: User-based project isolation
• Permission checking: Request-level authorization
• Multi-tenancy: Complete user data isolation

Configuration (src/core/config/)

The configuration system manages all runtime settings and environment-specific configurations.

Configuration Files

• development.json: Development environment settings
• production.json: Production environment settings

Configuration Sections

Server Configuration

{
  "server": {
    "port": 8042,
    "host": "localhost",
    "max_connections": 1000,
    "timeout": 30
  }
}

Database Configuration

{
  "database": {
    "data_path": "./data",
    "max_file_size": "100MB",
    "backup_enabled": true,
    "backup_interval": "24h"
  }
}

Security Configuration

{
  "security": {
    "jwt_secret": "your-secret-key",
    "jwt_expiration": "1h",
    "encryption_enabled": true,
    "rate_limiting": true
  }
}

Dynamic Configuration

• Runtime updates: Configuration changes without restart
• Environment detection: Automatic environment selection
• Validation: Configuration validation on startup

Projects (src/core/engine/projects/)

The projects component manages the top-level organizational structure and user isolation.

Project Lifecycle

• Creation: New project initialization
• Management: Project metadata and configuration
• Access control: User-based project ownership
• Cleanup: Project deletion and resource management

User Isolation

• Namespace separation: Each user has isolated project spaces
• Access verification: All operations verify project ownership
• Resource limits: Per-user resource quotas and limits

Library (src/library/)

The library component contains shared functionality, common types, and utility functions used across all components.

Common Types

• Data types: Definitions for all supported data types
• Error types: Standardized error handling structures
• Response types: Consistent API response formats

Utilities

• String operations: Common string manipulation functions
• Date/time handling: Date parsing and formatting
• JSON processing: Serialization and deserialization
• File operations: Safe file system interactions

Shared Functionality

• Logging: Centralized logging system
• Validation: Data validation functions
• Conversion: Type conversion utilities
• Constants: System-wide constants and defaults

Component Interactions

Request Flow

1. Server receives HTTP request
2. Configuration provides runtime settings
3. Security validates authentication and authorization
4. Engine processes the request
5. Data Layer performs storage operations
6. Projects manages user isolation
7. Library provides shared utilities throughout
8. Server returns formatted response

Data Flow

Client → Server → Security → Engine → Data Layer → Storage
                     ↓           ↑
               Configuration  Library

Development Guidelines

Adding New Components

1. Create component directory under appropriate namespace
2. Implement core functionality with proper error handling
3. Add configuration options if needed
4. Integrate with security layer for access control
5. Add comprehensive logging
6. Write tests for all functionality

Component Communication

• Use well-defined interfaces between components
• Implement proper error propagation
• Use library utilities for common operations
• Follow established patterns for data flow

Next Steps

Learn more about specific aspects:

• Data Structure - How data is organized
• API Reference - REST API endpoints and usage
• Security - Authentication and encryption details