Data Mirror | Real-Time Sync and CDC Python Script App

DATA MIRROR (v0.8)

The Ultimate Real-Time Pipeline for Syncing Your Relational Databases with Sinks or Custom Replica Databases.

Data Mirror is a production-grade, high-performance synchronization bridge. While it excels at bringing the power of sink engines (Meilisearch, Elasticsearch, OpenSearch) to your datasets, it is designed as a universal pipeline.

Whether you need to maintain a real-time replica database or sync with custom internal systems, Data Mirror handles the heavy lifting of Change Data Capture (CDC).

Important Cost Warning

While the open-source versions of these search engines are free to use, please be aware that deploying them on cloud infrastructure or utilizing managed third-party services (such as Meilisearch Cloud or Elasticsearch Service) will incur separate hosting and usage costs. You are responsible for monitoring and managing the expenses associated with your cloud deployments.

Complete Docs in Detail

Quick Setup

TOP KEY FEATURES

•	Instant Real-Time Sync: Your synchronized data stay updated the second your data changes. No delays, no stale information—just lightning-fast synchronization.

•	Millions of Records: Easily move millions of existing records into your sink engine on the first run. Ensures no data is ever missed.

•	Smart Relationships: Automatically converts separated but related data (like categories inside a product) into a single flat record, so your search or sink engine can read it instantly without complex queries.

•	Unlimited Flexibility: Native support for major engines, plus a robust plugin system. Universal pipeline: maintain a real-time replica DB, sync to a Cache, or pipe data to your own API with a simple Python plugin.

•	Effortless Deployment: One Docker Compose file and one simple schema file. Designed to be plug-and-play in any infrastructure.

•	Beautiful Monitoring: Keep an eye on everything through a simple dashboard. Track speeds and pipeline health at a glance.

WHY USE EXTERNAL SEARCH BACKEND?

Offload complex queries and maintain specialized data views without straining your primary production database.

1. Speed
• Standard Database (SQL): Slows down as data grows
• Search Backend: Millisecond responses at scale

2. Fuzzy Matching
• Standard Database (SQL): Extremely limited or slow
• Search Backend: Handles typos automatically

3. Ranking
• Standard Database (SQL): Hard to implement
• Search Backend: Sophisticated relevancy ranking

4. Indexing
• Standard Database (SQL): Heavy on DB resources
• Search Backend: Optimized for read-heavy workloads

REAL-TIME REPLICA WITHOUT THE HASSLE

Setting up database replication often involves complex triggers or expensive enterprise tools. Data Mirror simplifies this completely.

Cross-Format Sync

Sync a PostgreSQL source to a MySQL sink, or even a SQL source to a NoSQL target like MongoDB via custom plugins.

COMPATIBILITY & INTEGRATION

Use Your Existing Databases! You don’t need to migrate your data. Simply point the engine to your existing instance, and it monitors changes.

Source: PostgreSQL, MySQL Sink: Meilisearch, Elasticsearch, OpenSearch, Replica DBs, Custom Backends Infra: Docker (Kafka or Local Broker supported)

SYSTEM REQUIREMENTS

•	Docker and Docker Compose (Mandatory)

•	PostgreSQL (v12+) (Required based on usage)

•	MySQL (v8.0+) (Required based on usage)

•	A search engine (sink) – Meilisearch, Elasticsearch, or OpenSearch (Required based on usage)

•	Python (v3.9+) (Required based on usage)

EASY-TO-WRITE SCHEMA

Configuration is incredibly intuitive. Map tables, define relationships, and transform data in a simple YAML file.

schemas:
  - database: "store_db" 
    index: "products_index" 
    enabled: true
    table: "products" 
    primary_key: "id" 
    columns:
      - id
      - product_name
      - sku
      - price
    relationships:
      - name: "category" 
        type: "many_to_one" 
        table: "categories" 
        join_condition:
          parent_key: "category_id" 
          child_key: "id" 
        columns:
          - id
          - category_name
    transform:
      rename:
        product_name: "name" 
      cast:
        price: "float"

WHAT IS INCLUDED?

•	Docker Image: Ready-to-deploy, optimized sync engine.

•	Complete Documentation: Getting started guides and API specs.

•	Transformation Guide: Writing custom Python scripts for logic.

•	Plugin Architecture: Build your own Sink plugins.

•	Management Dashboard: Built-in web tools for monitoring.

MAKE YOUR PRODUCT UNSTOPPABLE

Data Mirror delivers the foundation for building high-quality, lightning-fast user experiences. Save weeks of work and scale your data synchronization capabilities today.