diff --git a/README.en.md b/README.en.md index b72a723a5ec60d47571404a752610c406dccd2a9..1d046f5dc142765486eca95b93aa8955d7380b9e 100644 --- a/README.en.md +++ b/README.en.md @@ -10,6 +10,17 @@ Full data verification: JDBC is used to extract the source and target data, and Incremental data verification, through debezium monitoring the data change records of the source database, the extraction service regularly processes the change records of debezium according to a certain frequency, and makes statistics on the change records. Send the statistical results to the data verification service. The data verification service initiates incremental data verification and outputs the verification results to the specified path file. + + +**Installation environment requirements:** + +``` +JDK11+ +Install kafka (start zookeeper and kafka service) +``` + + + #### Installation 1. Download and start Kafka @@ -69,32 +80,84 @@ transforms.Reroute.topic.replacement=data_check_test_all bin/connect-standalone -daemon etc/kafka/connect-standalone.properties etc/kafka/mysql-conect.properties ``` + **Verify service startup configuration** +``` +Verify the service configuration and modify the application.yml file + server.port is the verification service web port, which can not be modified by default + bootstrap-servers is the working address of kafka, and the default installation can not be modified + data.check.data-path is the output address of the verification result, and the default configuration can not be modified + data.check.source-uri the source side service request address, and the default configuration can not be modified + data.check.sink-uri is the service request address of the target end, and the default configuration can not be modified +``` -**Start datachecker performance service** + **Source side service startup configuration** ``` -Source side extraction service -java -jar datachecker-extract.jar -Dspring.config.additional-location=.\config\application-source.yml +Source side service configuration modification application-source.yml file + server.port is the source side extraction service web port, which can not be modified by default + spring.check.server-uri is the verification service request address, and the default configuration can not be modified + spring.extract.schema is the current validation data schema, and the name of the MySQL database + bootstrap-servers is the working address of kafka, which can not be modified by default + + Data Source Configuration +``` -Destination extraction service -java -jar datachecker-extract.jar -Dspring.config.additional-location=.\config\application-sink.yml + **Target side service startup configuration** -or use extract-endpoints shell command to start the source and sink service -sh extract-endpoints.sh start|stop|restart +``` +Target side service configuration modification application-sink.yml file + server.port is the sink side extraction service web port, which can not be modified by default + spring.check.server-uri is the verification service request address, and the default configuration can not be modified + spring.extract.schema is the current validation data schema, and the name of the MySQL database + bootstrap-servers is the working address of kafka, which can not be modified by default + + Data Source Configuration +``` -check service -java -jar datachecker-check.jar -Dspring.config.additional-location=.\config\application.yml -or use check-endpoint shell command to start the check service +**Start datachecker performance service** + +``` +use extract-endpoints shell command to start the source and sink service +sh extract-endpoints.sh start|stop|restart +use check-endpoint shell command to start the check service sh check-endpoint.sh start|stop|restart + +The extraction service must be started first, and then the verification service. +``` + + **Background start command** + +``` +nohup java -Dspring.config.additional-location=config/application-source.yml -jar datachecker-extract-0.0.1.jar --spring.profiles.active=source >/dev/null 2>&1 & + +nohup java -Dspring.config.additional-location=config/application-sink.yml -jar datachecker-extract-0.0.1.jar --spring.profiles.active=sink >/dev/null 2>&1 & + +nohup java -Dspring.config.additional-location=config/application.yml -jar datachecker-check-0.0.1.jar >/dev/null 2>&1 & +``` + + + +**After the verification service is fully started, a verification request is sent** + +``` +curl -X 'POST' 'http://localhost:9000/start/check?checkMode=FULL' -H 'accept: */*' -d '' -H 'Content-Type: application/json' + +Note: localhost: 9000, which is the verification service request address, and port 9000 is the default port set for the verification service ``` + + **remarks: ** ``` -The incremental verification service is started, and the source side configuration file config\application-source.yml needs to be modified. -debezium-enable:true -And configure other debezium related configurations. The incremental verification service can be started when the service is started +1. Single instance verification uses sh script to start the verification service. If verification needs to be started in parallel, copy the current working directory file. After reconfiguration, use the java background startup command. + +2. After the extraction service is started, it will automatically load the table related information of the database. If the data volume is large, the data loading will be time-consuming. + +3. After the validation service is started, it will detect whether the table data information on the extraction end has been loaded. If the loading is not completed within a certain period of time, the validation service will automatically exit. At this time, you need to query the table information loading progress of the source and destination, and view the loading progress through the log information. Or restart the verification service directly. + +4. The incremental verification service is started, and the source side configuration file config application source needs to be modified Debezium enable: true in yml and configure other debezium related configurations. Start the service to start the incremental verification service ``` diff --git a/README.md b/README.md index 99ba3f9a4e989e399ba61c2526f193a4f8cc1e16..8e9a24bb1bb7a4bf902f45c947f4211a7ebdedbf 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,10 @@ openGauss数据迁移校验工具 ，包含全量数据校验以及增量数据 增量数据校验，通过debezium监控源端数据库的数据变更记录，抽取服务按照一定的频率定期处理debezium的变更记录，对变更记录进行统计。将统计结果发送给数据校验服务。由数据校验服务发起增量数据校验，并将校验结果输出到指定路径文件。 +**安装环境要求：** + + JDK11+ + kafka安装（启动zookeeper，kafka服务） #### 安装教程 @@ -15,7 +19,7 @@ openGauss数据迁移校验工具 ，包含全量数据校验以及增量数据 2. 获取数据校验服务jar包，及配置文件模版（datachecker-check.jar/datachecker-extract.jar,application.yml,application-sink.yml,application-source.yml） 3. 将jar包以及配置文件copy到指定服务器目录，并配置相关配置文件，启动相应的jar服务即可。 -#### 使用说明 +#### 详细使用说明 **启动Zookeeper** @@ -64,30 +68,80 @@ transforms.Reroute.topic.replacement=data_check_test_all bin/connect-standalone -daemon etc/kafka/connect-standalone.properties etc/kafka/mysql-conect.properties ``` +**校验服务启动配置** + +``` +校验服务配置 修改application.yml文件 + server.port 为校验服务web端口，默认可不修改 + bootstrap-servers 为kafka工作地址，默认安装可不修改 + data.check.data-path 校验结果输出地址，默认配置可不修改 + data.check.source-uri 源端服务请求地址，默认配置可不修改 + data.check.sink-uri 目标端服务请求地址，默认配置可不修改 + data.check.core-pool-size 并发线程数设置，根据当前环境配置，可不修改 +``` + +**源端服务启动配置** + +``` +源端服务配置 修改application-source.yml文件 + server.port 为源端抽取服务web端口，默认可不修改 + spring.check.server-uri 校验服务请求地址，默认配置可不修改 + spring.extract.schema 当前校验数据schema，mysql 数据库名称 + spring.extract.core-pool-size 并发线程数设置，根据当前环境配置，可不修改 + bootstrap-servers 为kafka工作地址，默认安装可不修改 + + 数据源配置 + +``` + +**目标端服务启动配置** + +``` +目标端服务配置 修改application-sink.yml文件 + server.port 为目标端抽取服务web端口，默认可不修改 + spring.check.server-uri 校验服务请求地址，默认配置可不修改 + spring.extract.schema 当前校验数据schema，opengauss schema名称 + spring.extract.core-pool-size 并发线程数设置，根据当前环境配置，可不修改 + bootstrap-servers 为kafka工作地址，默认安装可不修改 + + 数据源配置 +``` + + + **启动数据校验服务** +```shell +sh extract-endpoints.sh start|restart|stop +sh check-endpoint.sh start|restart|stop +先启动抽取服务，后启动校验服务。 ``` -#源端抽取服务 -java -jar datachecker-extract.jar -Dspring.config.additional-location=.\config\application-source.yml -#宿端抽取服务 -java -jar datachecker-extract.jar -Dspring.config.additional-location=.\config\application-sink.yml +**后台启动命令** -或者使用 -sh extract-endpoints.sh start|stop|restart 命令 +```shell +nohup java -Dspring.config.additional-location=config/application-source.yml -jar datachecker-extract-0.0.1.jar --spring.profiles.active=source >/dev/null 2>&1 & -校验服务 -java -jar datachecker-check.jar -Dspring.config.additional-location=.\config\application.yml -或者使用 -sh check-endpoint.sh start|stop|restart 命令 +nohup java -Dspring.config.additional-location=config/application-sink.yml -jar datachecker-extract-0.0.1.jar --spring.profiles.active=sink >/dev/null 2>&1 & + +nohup java -Dspring.config.additional-location=config/application.yml -jar datachecker-check-0.0.1.jar >/dev/null 2>&1 & +``` + +**校验服务完全启动成功后，发起校验请求。** + +```shell +curl -X 'POST' 'http://localhost:9000/start/check?checkMode=FULL' -H 'accept: */*' -d '' -H 'Content-Type: application/json' + +说明：localhost:9000，为校验服务请求地址，端口9000为校验服务默认设置端口 ``` -备注： +**备注：** ``` -增量校验服务启动，需要修改源端配置文件\config\application-source.yml -debezium-enable:true -并配置其他 debezium相关配置，服务启动即可开启增量校验服务 +1、单实例校验使用sh 脚本启动校验服务，如果需要并行开启校验，复制当前工作目录文件，重新配置后，使用java 后台启动命令。 +2、抽取服务在启动后，会自动加载数据库的表相关信息，如果数据量较大，则数据加载会比较耗时。 +3、校验服务启动后，会检测抽取端的表数据信息是否加载完成，如果在一定时间内，未完成加载，则校验服务会自行退出。这时需要查询源端和宿端的表信息加载进度，通过日志信息查看加载进度。或者直接重新启动校验服务。 +4、增量校验服务启动，需要修改源端配置文件\config\application-source.yml 中 debezium-enable:true并配置其他 debezium相关配置，服务启动即可开启增量校验服务 ``` **开发人员本地 启动服务** diff --git a/config/application-sink.yml b/config/application-sink.yml index 39b700a0d5e0217f3112a73289f95d224d521a43..f07c855138af1b3c94f1513a7eb275940a556969 100644 --- a/config/application-sink.yml +++ b/config/application-sink.yml @@ -12,6 +12,6 @@ spring: druid: dataSourceOne: driver-class-name: org.opengauss.Driver - url: jdbc:opengauss://{ip}:{port}/{schema_db}?useSSL=false&useUnicode=true&characterEncoding=utf-8&serverTimezone=UTC + url: jdbc:opengauss://127.0.0.1:5432/postgres?useSSL=false&useUnicode=true&characterEncoding=utf-8&serverTimezone=UTC username: password: 'xxxx' # The password text may contain special characters, which need to be enclosed in quotation marks \ No newline at end of file diff --git a/config/application-source.yml b/config/application-source.yml index ee542138137b48bedf65b66cb41dd10f0eeaaeab..70e0f3f04ac832e9f4aaee81149bb77f13897c2e 100644 --- a/config/application-source.yml +++ b/config/application-source.yml @@ -20,6 +20,6 @@ spring: druid: dataSourceOne: driver-class-name: com.mysql.cj.jdbc.Driver - url: jdbc:mysql://{ip}:{port}/{database}?useSSL=false&useUnicode=true&characterEncoding=utf-8&serverTimezone=UTC&allowPublicKeyRetrieval=true + url: jdbc:mysql://127.0.0.1:3306/mysql?useSSL=false&useUnicode=true&characterEncoding=utf-8&serverTimezone=UTC&allowPublicKeyRetrieval=true username: password: 'xxxx' # The password text may contain special characters, which need to be enclosed in quotation marks \ No newline at end of file