Big Data Collection and Analysis Solutions

Aggregation, Heterogeneous, Consistency

Download Materials

supports synchronous multi-source heterogeneous database synchronization;

supports remote deployment with the ability to deploy remote nodes offline;

supports data comparison and repair to ensure data consistency;

possesses the capability for visualizing the synchronization process.

Solution Background

In the current era of the Internet, big data, and artificial intelligence technologies, data plays an increasingly crucial role. Data now has various requirements such as mobility, aggregation, and migration to the cloud. Consequently, data within databases has demands for heterogeneous migration, heterogeneous synchronous replication, and aggregating data onto big data platforms. The mobility and aggregation of data enable enterprises' businesses and applications to break through original data barriers and conduct deeper analysis of existing data to obtain comprehensive data-driven decisions.

With the advent of the big data era, more and more industries are recognizing the importance of big data analysis. Systems such as transaction log monitoring and analysis, abnormal transaction warning systems, and customer profiling systems all leverage big data analysis to mine data and realize the appreciation of data value. Big data analysis is currently a hot topic in industry development, and its foundation is the accumulation of a large amount of reliable data. Data ingestion middleware can provide big data analysis platforms with continuous, reliable data sourced from actual business operations, delivering high-performance, fault-tolerant, easy-to-use, and flexible real-time data streams suitable for big data environments. This helps clients expand real-time data integration architectures into big data systems without compromising production system performance.

Solution Overview

Both i2Stream and i2Active operate on the same principle, utilizing database semantic-level real-time data replication software to achieve full and incremental synchronization of databases in high-concurrency transaction scenarios. Through synchronization verification, they ensure transactional-level eventual consistency between the source and target ends of the database; the i2Stream product is designed for heterogeneous data source scenarios, assisting users in completing tasks such as heterogeneous data migration, data distribution, and data warehouse construction in complex application environments.

Data Ingestion: Capture data from numerous data sources, including databases, semi-structured files, etc.

Data Migration to Cloud: Captured data supports hybrid cloud scenarios, enabling interoperability between on-premises and cloud environments.

Data Processing: Possesses capabilities for data cleaning, transformation, and analysis.

Solution Design

Overall Design Concept of the Solution

Implementation of Database Extraction and Collection

i2Stream extracts data from multiple local data sources, aggregates it into a cloud-based data warehouse, and ensures data integrity and consistency.

i2Stream supports multi-dimensional data report statistics and can display data synchronization status in near real-time, ensuring data transparency and validity.

To prevent data loss, i2Stream provides a protection mechanism for converting physical deletions to insertions. When a record in the production database is physically deleted, this record can be inserted into other corresponding databases, allowing reverse synchronization when the production database needs to be restored.

The design of the data collection architecture references a model (as shown in the diagram below) for the overall architecture design.

For different databases, data is extracted and transformed based on write-ahead logging and sent to various target ends, including message queues, databases, big data platforms, and files. The data transmission process is divided into full and incremental types. Full data is obtained through logical extraction from the database. Incremental data is obtained through log analysis. Full and incremental data to secondary databases can achieve continuous data replication, completing real-time protection of database data. Full and incremental data replicated to other target ends through synchronization software constitutes heterogeneous replication.

Source databases support traditional commercial databases such as Oracle, DB2, SQL Server, as well as domestic databases like Dameng, GaussDB, TDSQL, OB, etc. Heterogeneous target ends support various data warehouses, message queues, MPP, NoSQL, etc.

Solution Advantages

i2Stream supports synchronization of multiple heterogeneous databases, allowing data from various databases to be synchronized to the target database. The architecture supports standalone, master-slave, and distributed configurations.

i2Stream supports remote deployment with the ability to deploy remote nodes offline, supporting hybrid cloud scenarios.

i2Stream supports data comparison and repair to ensure data consistency.

i2Stream possesses the ability to visualize the synchronization process, enabling users to intuitively grasp the data situation.

Appication Scenarios

multi-node data concentration

1. Multi-node Data Aggregation

a. High-performance end-to-end data integration

b. Establishing a centralized data center

2. Advantages

a. Supports heterogeneous hardware and operating environments

b. Supports different database versions

c. Supports local or remote deployment

d. Implementation is installation-free, fully based on a B/S architecture

Customer Case Study

Case Name: Sichuan Provincial Medical Insurance Bureau

Case Description

The Sichuan Provincial Medical Insurance Bureau is a newly established unit following the institutional reform in Sichuan Province in 2018. This project involved migrating the medical insurance data of 22 prefectures and cities in Sichuan Province, covering 83 million people across the province, with approximately 100TB of data migrated to the cloud. It involved various types of data synchronization, including real-time and scheduled data synchronization, as well as homogeneous and heterogeneous database synchronization; the databases involved include Oracle, MySQL, PostgreSQL, TDSQL, TBASE, etc.

Case Value

1. Aggregated multi-source data in a hybrid cloud scenario

2. Delete-to-Insert conversion suited for data organization and data asset management scenarios