Skip to main content

Establishing Reliable Edge-to-Cloud Data Synchronization

This guide explains how to configure MonoTS to push locally persisted table changes to central platforms using built-in Change Data Capture (CDC) streams.

MonoTS acts as a durable, ordered local buffer in front of your central streaming or analytical platform, retrying through network outages using server-side checkpoints.

Supported architectures​

Depending on your sink type, MonoTS supports different synchronization patterns:

A. Real-time row sync to message queues (Apache Kafka)​

Best for downstream real-time stream processing (for example Flink or Spark Streaming). MonoTS pushes row events in JSON format.

[Sensors] --SQL INSERT--> [MonoTS Edge] --(PUSH JSON)--> [Cloud Kafka]

B. Batch sync to data lakes (Delta Lake / filesystem)​

Best for historical analytics and archiving. MonoTS synchronizes Parquet files and maintains transaction logs.

[Sensors] --SQL INSERT--> [MonoTS Edge] --(PUSH Parquet)--> [S3 / MinIO (Delta Lake)]

Resilience and outage behavior​

MonoTS guarantees at-least-once delivery to sinks, even over shaky WAN links:

  1. Local persistence: Edge inserts keep succeeding into MonoTS local storage during WAN outages.
  2. Sync pause: The background CDC stream detects connectivity failure and enters a retry loop. It does not block local ingestion.
  3. Automatic resume: Upon reconnection, the stream resumes from the last acknowledged checkpoint (acked_lsn), so you do not lose progress.
  4. Consumer idempotency: Because delivery is at-least-once, downstream consumers should tolerate duplicates.

CDC modes​

Stream property cdc.mode supports:

ModeBehavior
batchExport sealed Parquet / historical files (default for filesystem and Delta). Memtable data needs FLUSH before it appears.
hybridHistorical files plus live WAL tailing (default for Kafka).

Configuration examples​

Use the CREATE STREAM SQL command to manage synchronization. Start from a table created in the Quick Start.

Example 1: Synchronizing to Apache Kafka​

CREATE STREAM edge_to_cloud_kafka WITH (
  'sink.type' = 'kafka',
  'sink.kafka.brokers' = 'cloud-kafka-broker:9092', -- comma-separated
  'sink.kafka.topic' = 'incoming-edge-data',
  'source.table' = 'local_temp_sensor'
);
note

Kafka sinks default to cdc.mode = hybrid (historical backfill + live tailing). The only supported Kafka message format today is JSON.

Example 2: Synchronizing to Delta Lake (S3-compatible)​

CREATE STREAM edge_to_datalake_delta WITH (
  'sink.type' = 'delta',
  'source.table' = 'local_temp_sensor',
  'sink.delta.path' = 's3://my-bucket/datalake/sensor_data',
  'sink.delta.endpoint' = 'https://minio.mycorp.com:9000' -- optional S3 endpoint
);
note

Delta sinks default to cdc.mode = batch (synchronizing sealed Parquet files). Configure S3 credentials via AWS_* environment variables on the monots-server host.

Example 3: Synchronizing to a local filesystem​

CREATE STREAM edge_to_fs WITH (
  'sink.type' = 'filesystem',
  'sink.filesystem.path' = '/tmp/monots-export/sensor',
  'source.table' = 'local_temp_sensor'
);

Stream management​

SHOW STREAMS;
SHOW STREAM STATUS FOR edge_to_cloud_kafka; -- Monitor acked_lsn, phase
DROP STREAM edge_to_cloud_kafka;
Note on SQL FLUSH

For batch-mode sinks (Delta Lake or filesystem), rows currently in the active memtable are not synchronized until they are sealed into a Parquet file, either by automatic memory limits or a manual FLUSH TABLE command.

Unsupported options​

Do not document or rely on:

  • sink.format = 'arrow' for Kafka (JSON only today)
  • HTTP curl to port 8080 for SQL