Bar Snir

Using the right tools: 1. Does the throughput & volume justify big data tools? Distributed systems today are designed to support big data volumes, and they most efficient On a larger scale ( resource balancing, high availability, etc…) If your data comes in small throughput you might don't need to over-engineer your system design. 2. Can you offer your business a nearly real-time solution? If the answer is yes, watch for Spark structured streaming in combination with Apache Kafka. 3. If you need real-time processing and you going to deal with large throughput….consider the combination of Apache Kafka with Apache Flink…the most powerful combination for real-time processing that I have been experienced for so long.

1. Plan your partition distribution: Small partitions, with low data volume, can cost more resources to query. Please calculate your partition size and adjust your processor parallelism. 2. Monitor your ETL data quality. How much Null do you receive in your nullable column? If you monitor the process with proper tools (for example Prometheus & Grafna), you might find interesting behavior in your data. 3. Design well operate jobs: For example, design your code with session configuration that can be provided from the job’s arguments (outer arguments). Control your job without code editing. 4. Optimize a number of machine in Dev env: Try to simulate as possible production env, for getting the correct machine specs for the job.

Apache Flink is a stream-first processor. 1. Kafka friendly: Including insert or upsert connectors, for Apache Kafka, Flink has a built-in native configuration for Kafka’s producer and consumer. This fact leads to high-performance real-time processing, with Flink table API or SQL. 2. Confluent’s schema registry friendly: Include supporting JSON, Avro, and more, Flink also supports the extraction of Debezium CDC messages by updating the state tables with proper operation (insert, update, delete)… which saves a lot of implementation. Also…the connectors have fully supported communication with the schema registry, even with a secured one. There are more points why it's stream-friendly, check out this tech for streaming processing.

In my experience, when it comes to No SQL data sources, the bottleneck can be in large update volumes. Some points can help us live well with these updates: 1. Alter the document: avoiding partial updates and replacing the entire document in most cases are more acceptable. Updating the partial fields are more heavy process for segments and filesystem. 2. Use defined bulk size: with the proper metrics and the time diff ingest the data in a way the system can handle it, without causing damage to production, such as search or read queries. 3. The magic trick for updating all the documents : create processes & operations that make an up-to-date collection, and make the read services point to the up-to-date collection.