Kalyani Deshmukh

🌟𝐖𝐞'𝐫𝐞 𝐇𝐢𝐫𝐢𝐧𝐠: 𝐏𝐲𝐭𝐡𝐨𝐧 𝐀𝐏𝐈 + 𝐄𝐥𝐚𝐬𝐭𝐢𝐜𝐬𝐞𝐚𝐫𝐜𝐡 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐫𝐬! TekPillar®🌟 🔧𝐏𝐨𝐬𝐢𝐭𝐢𝐨𝐧 : Python API+ Elastic Search 📍𝐋𝐨𝐜𝐚𝐭𝐢𝐨𝐧 : Bangalore  📅𝐄𝐱𝐩𝐞𝐫𝐢𝐞𝐧𝐜𝐞 : 5+ Years      #KeyResponsibilities: 👉Develop custom search and analytics applications using FastAPI and Elasticsearch. 👉Index data from various sources, including big data systems like Databricks and Snowflake. 👉Query Elasticsearch data and create robust REST APIs. Implement AI/ML use cases and optimize Elasticsearch performance. 👉Create reusable libraries for index templates, custom analyzers, and scripts. 👉Collaborate with data engineers on pipelines and ETL jobs. 👉Guide developers on search, filtering, and aggregation requirements. 👉Research and prototype new Elasticsearch features. 🔗 𝐀𝐩𝐩𝐥𝐲 𝐇𝐞𝐫𝐞:  📞8799447189 | ✉️ [email protected] #Python #Elasticsearch #FastAPI #BigData #Azure #TechJobs #Bangalore #Hiring #tekpillarjobs

5

Real-Time Indexing at Scale: How Meesho Leveraged Apache Flink 🚀 At Meesho, they support millions of entrepreneurs across Bharat. With over 50 million products in their catalog, keeping product data fresh in real-time is essential. Delays in indexing lead to users seeing outdated prices or out-of-stock items, causing frustration, cart abandonment, and potential customer loss 😯 Here's how we overhauled our indexing architecture using Apache Flink for horizontal scalability 👇🏻 They started with a crontab-based approach, scheduling indexing jobs every 10 minutes. It worked well initially, with indexing taking 5–7 minutes for our smaller product base 🕑 The Scaling Challenge 📈 As Meesho grew, so did their problems 👇🏻 1️⃣ Exponential Growth: The number of products exploded into tens of millions, pushing indexing time to 4 hours. 2️⃣ Data Lag: Users saw outdated prices and stock information, especially during peak sales. 3️⃣ Processing Inefficiency: Recomputing all product data—even if unchanged—added unnecessary load. 4️⃣ Failure Delays: Job failures caused delays of up to 8 hours. Interim Fix: Kafka Consumers ✅ While plotting a long-term solution, they implemented Kafka consumers to 👇🏻 1️⃣ Parallelize Elasticsearch Writes: This reduced ES write time from 2 hours to 15 minutes, cutting overall indexing time to ~3 hours. 2️⃣ Adopt Batch Processing: Catalogs were sent directly to Kafka for immediate indexing, reducing memory consumption. Though this helped temporarily, as their catalog grew, indexing time soon crept back up to 4 hours. Requirements for the New Solution ✍🏻 They needed a system that was 👇🏻 1️⃣ Low-latency & fault-tolerant: Processing events in seconds, not hours. 2️⃣ Decoupled: Separating indexing from the enrichment pipeline. 3️⃣ Extensible: Supporting multi-consumer queues and future data flow operations. Why They Chose Apache Flink ❓ After evaluating several options, Apache Flink stood out for 👇🏻 1️⃣ Exactly-once processing guarantees: Ensuring data consistency. 2️⃣ Very low latency: Even with fault tolerance enabled. 3️⃣ High throughput: Capable of processing 182 million records/sec across 30 nodes. Flink vs. Storm: Performance Benchmark 😎 1️⃣ Flink (no fault tolerance): 182M records/sec, 0 ms latency. 2️⃣ Flink (with 5 sec checkpoints): 178M records/sec, 0 ms latency. 3️⃣ Storm (no fault tolerance): 9.8M records/sec, 11 ms latency. 4️⃣ Storm (with fault tolerance): 0.57M records/sec, 30–120 ms latency. 5️⃣ Flink had 300× higher throughput than Storm, even with fault tolerance. ✅ Read Full blog - https://lnkd.in/gQS_WyQt 🚨 Join my high quality, industry & modern tech stack based and practical project driven Data Engineering BootCAMP✌🏻👇 👉 Enroll Here - https://bit.ly/4eA2tuX 🚀 Dedicated Doubt Support & Placement Assistance 📲 Call/WhatsApp for any query (+91) 9893181542 Cheers - Grow Data Skills 🙂 #dataengineering

78

2 Comments

𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐢𝐬 𝐭𝐡𝐞 𝐏𝐲𝐭𝐡𝐨𝐧 𝐀𝐏𝐈 for Apache Spark, designed for large-scale data processing. It's known for: 𝐒𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: Handling massive datasets across distributed computing clusters. 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: Optimized for high-performance data processing with features like in-memory computing. 𝐁𝐢𝐠 𝐃𝐚𝐭𝐚 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧: Seamlessly integrates with big data frameworks and tools. 𝐁𝐞𝐬𝐭 𝐟𝐨𝐫: Large-scale data processing tasks that require distributed computing and efficient handling of vast amounts of data. 𝐏𝐚𝐧𝐝𝐚𝐬 is a go-to library for data manipulation and analysis in Python. It's widely used for: 𝐃𝐚𝐭𝐚𝐅𝐫𝐚𝐦𝐞 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬: Efficiently handling data in-memory with powerful data manipulation capabilities. 𝐑𝐢𝐜𝐡 𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧𝐚𝐥𝐢𝐭𝐲: Comprehensive functions for data cleaning, transformation, and aggregation. 𝐄𝐚𝐬𝐞 𝐨𝐟 𝐔𝐬𝐞: Intuitive API and excellent support for smaller datasets. Best for: Small to medium-sized datasets that fit into memory and require intricate data manipulation. 𝐖𝐡𝐞𝐧 𝐭𝐨 𝐂𝐡𝐨𝐨𝐬𝐞 𝐖𝐡𝐢𝐜𝐡? For Smaller, In-Memory Data Tasks: Use Pandas for its ease of use and rich functionality. For Large-Scale Data Processing: Opt for PySpark to leverage its distributed computing power and scalability. doc credit - Shwetank Singh #aws #dataengineering ✅ Enrollments for Version 3 of scenario based structured hands-on Real-world AWS Data Engineering ( RADE™) program 𝐖𝐈𝐋𝐋 𝐂𝐋𝐎𝐒𝐄 𝐕𝐄𝐑𝐘 𝐒𝐎𝐎𝐍! 𝑵𝒆𝒘 𝑨𝑾𝑺 𝑫𝒂𝒕𝒂 𝑬𝒏𝒈𝒊𝒏𝒆𝒆𝒓𝒊𝒏𝒈 𝑩𝒂𝒕𝒄𝒉 𝒕𝒐 𝒔𝒕𝒂𝒓𝒕 𝑱𝒖𝒍𝒚 28𝒕𝒉- 𝐉𝐨𝐢𝐧 𝐚𝐥𝐨𝐧𝐠 𝐰𝐢𝐭𝐡 60 𝐚𝐦𝐚𝐳𝐢𝐧𝐠 𝐩𝐞𝐨𝐩𝐥𝐞 𝐰𝐡𝐨 𝐡𝐚𝐯𝐞 𝐞𝐧𝐫𝐨𝐥𝐥𝐞𝐝. ✅𝐈𝐟 𝐲𝐨𝐮 𝐚𝐫𝐞 𝐚𝐜𝐭𝐢𝐨𝐧-𝐨𝐫𝐢𝐞𝐧𝐭𝐞𝐝, 𝐫𝐞𝐠𝐢𝐬𝐭𝐞𝐫 𝐍𝐎𝐖 for FREE 𝐟𝐨𝐫 𝐭𝐡𝐞 𝐦𝐚𝐬𝐭𝐞𝐫𝐜𝐥𝐚𝐬𝐬- 𝐥𝐨𝐨𝐤 𝐚𝐭 𝐜𝐨𝐦𝐦𝐞𝐧𝐭 𝐛𝐞𝐥𝐨𝐰!

143

5 Comments

𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐢𝐬 𝐭𝐡𝐞 𝐏𝐲𝐭𝐡𝐨𝐧 𝐀𝐏𝐈 for Apache Spark, designed for large-scale data processing. It's known for: 𝐒𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: Handling massive datasets across distributed computing clusters. 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: Optimized for high-performance data processing with features like in-memory computing. 𝐁𝐢𝐠 𝐃𝐚𝐭𝐚 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧: Seamlessly integrates with big data frameworks and tools. 𝐁𝐞𝐬𝐭 𝐟𝐨𝐫: Large-scale data processing tasks that require distributed computing and efficient handling of vast amounts of data. 𝐏𝐚𝐧𝐝𝐚𝐬 is a go-to library for data manipulation and analysis in Python. It's widely used for: 𝐃𝐚𝐭𝐚𝐅𝐫𝐚𝐦𝐞 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬: Efficiently handling data in-memory with powerful data manipulation capabilities. 𝐑𝐢𝐜𝐡 𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧𝐚𝐥𝐢𝐭𝐲: Comprehensive functions for data cleaning, transformation, and aggregation. 𝐄𝐚𝐬𝐞 𝐨𝐟 𝐔𝐬𝐞: Intuitive API and excellent support for smaller datasets. Best for: Small to medium-sized datasets that fit into memory and require intricate data manipulation. 𝐖𝐡𝐞𝐧 𝐭𝐨 𝐂𝐡𝐨𝐨𝐬𝐞 𝐖𝐡𝐢𝐜𝐡? For Smaller, In-Memory Data Tasks: Use Pandas for its ease of use and rich functionality. For Large-Scale Data Processing: Opt for PySpark to leverage its distributed computing power and scalability. P.S ✅Version 3 of 𝐬𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝐛𝐚𝐬𝐞𝐝 𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐝 𝐡𝐚𝐧𝐝𝐬-𝐨𝐧 𝐑𝐞𝐚𝐥-world AWS Data Engineering ( RADE™) program 𝐰𝐢𝐭𝐡 80+ 𝐚𝐦𝐚𝐳𝐢𝐧𝐠 𝐩𝐞𝐨𝐩𝐥𝐞 completed on Oct 5th. Onboarding for the Next one to begin in November. ✅𝐈𝐟 𝐲𝐨𝐮 𝐚𝐫𝐞 𝐚𝐜𝐭𝐢𝐨𝐧-𝐨𝐫𝐢𝐞𝐧𝐭𝐞𝐝 and want to get in the next batch, fill the form below to be get invited for the webinar for RADE™ program V4 Click https://lnkd.in/ervbN-zt 𝐔𝐒𝐏: 𝐑𝐞𝐚𝐥-𝐰𝐨𝐫𝐥𝐝 𝐬𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝐛𝐚𝐬𝐞𝐝 𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐝 𝐡𝐚𝐧𝐝𝐬-𝐨𝐧 program ! doc credit - Shwetank Singh #aws #dataengineering

163

22 Comments

Maximize your PostgreSQL performance with this practical eBook! 🚀 Packed with real-world strategies, it covers query optimization, indexing, partitioning, schema design, and more. 💡 Learn how to optimize performance, improve monitoring, and lower infrastructure costs. Whether you're optimizing for speed or scalability, this guide is your go-to resource for better database management. https://bit.ly/4bL6bAv #PostgreSQL #OpenSource #database

3

Broadcast Join in PySpark 🈴 ================== A broadcast join sends the smaller table (or DataFrame) to all worker nodes, ensuring each worker node has a complete copy of the smaller table in memory.💯 A more detailed explanation of a broadcast join: 👇👇 1️⃣The large DataFrame is the one that contains a substantial amount of data. 2️⃣The small DataFrame is the one with significantly fewer rows or a smaller size. Gains because of a broadcast join: ❇️ 1️⃣Since the smaller DataFrame is available on each node, the join operation can be performed locally on each node without the need for extensive data movement. 2️⃣This reduces the amount of network traffic and can significantly improve the performance of the join, especially when the small DataFrame fits in the memory of each node. Tags: 🏛️ #data #dataengineering #databricks #dataengineerjobs #python #pyspark #bigdata #hadoopadmin #scala #dataanalytics #snowflake #snowflakedevelopers #datacloud #aws #awscloud #kafka #knowledgesharing #dailypost #datascience #machinelearning #AWS

14

🚀 Simplifying Data Ingestion with Databricks Autoloader 🚀 If you're dealing with large volumes of data, real-time ingestion can be a challenge. Databricks Autoloader simplifies this by automatically detecting new files in cloud storage and efficiently loading them into Delta Lake tables. 📌 Example: Let's say your data is continuously stored in a cloud directory. With Autoloader, you can ingest new files without worrying about complex batch processing. Here's a quick code example in Python: df = (spark.readStream .format("cloudFiles") .option("cloudFiles.format", "json") .option("cloudFiles.schemaLocation", "s3://your-bucket/schema-location/") .load("s3://your-bucket/data/")) df_processed = df.select("name", "age") df_processed.writeStream .format("delta") .outputMode("append") .option("checkpointLocation", "s3://your-bucket/checkpoints/") .table("people_data") 🔑 Key Benefits: 1) Automatic File Discovery 2) Schema Inference 3) Real-time Streaming With Databricks Autoloader, build scalable, low-latency data pipelines effortlessly! #Databricks #Autoloader #DataEngineering #DeltaLake #StreamingData #BigData

11

Use Cases and Best Practices Data quality is the backbone of reliable analytics, especially in big data environments. I recently explored methods to manage null values and outliers, which can significantly impact results if not addressed. Using PySpark’s.fillna() and .dropna() functions has been useful for scalable transformations. What are some of your go-to strategies for ensuring high data quality? #dataengineering #seniormanagers #dataarchitect #bigdata #databricks #dataquality #datacleaning #dataprofiling #datavalidation

5

Glad to hear 🤟 My Spark Streaming playlist on YouTube just helped someone to crack his JOB INTERVIEW 🤩 Congratulations Ravi Prakash 🏆 If you are reading this post, please tag yourself in the comments 😄 If you want to learn PySpark or Spark Streaming from basics to advanced Performance Optimization, check out the links in comments 👇 #spark #streaming #dataengineering

34

1 Comment

🌟 Actively Seeking New Job Opportunities! 🌟 Highly experienced Sr. Azure Data Engineer with 15+ years of expertise in designing and implementing scalable cloud data solutions, specializing in Azure, Azure Databricks, PySpark, and Python. Proficient in developing end-to-end ETL pipelines, data migration, and advanced analytics solutions, I have successfully led and executed cloud transformations across multi-cloud environments, including GCP and AWS. 🔑 Key Skills and Expertise: Azure Cloud Expertise: Extensive experience in building data solutions on Azure Synapse Analytics, Azure SQL Database, and Azure Data Factory (ADF) for large-scale data ingestion and transformation. Skilled in implementing Azure Databricks for high-performance data processing using PySpark and Python to manage complex business logic and deliver scalable data transformation pipelines. Advanced Analytics and Machine Learning: Developed and deployed machine learning models using Azure ML Studio and integrated with Databricks for advanced data analysis. Hands-on experience with MLlib and BERT models for predictive analytics, customer segmentation, and content recommendations. GCP Expertise: Led the migration of large datasets from Snowflake to Google Cloud Platform (GCP), leveraging BigQuery for efficient data warehousing, Google Cloud Storage (GCS) for scalable storage, and Google Cloud Composer for orchestrating data workflows. Implemented delta load and incremental update mechanisms using Cloud Dataflow and BigQuery MERGE operations. Data Governance and Security: Designed Data Clean Room (DCR) solutions on GCP and Azure, ensuring compliance with HIPAA and GDPR by implementing Google Cloud DLP, column-level encryption, and secure access controls using Azure Active Directory (AAD) and BigQuery Authorized Views. Cross-Cloud Integration: Expert in integrating Azure with GCP and AWS, developing hybrid cloud architectures, and utilizing Azure Data Lake Storage (ADLS Gen2), AWS S3, and GCS for seamless data migration and storage management. I am passionate about leveraging cloud technologies to drive business value through optimized data workflows, cutting-edge analytics, and secure, scalable architectures. Let’s connect to discuss how I can help your organization achieve its cloud data engineering goals. 📩 Feel free to reach out at [email protected] or via LinkedIn for new opportunities. #Azure #Databricks #PySpark #BigData #DataEngineering #GCP #MachineLearning #CloudComputing #AzureML #BigQuery #DataMigration

12

2 Comments

In an ideal world, we should not have the concept of warehouses in Snowflake. I don’t think I have ever met a data engineer who wants to delve deep into warehouse internals when they are already juggling tasks from different business stakeholders. Ideally, warehouses in Snowflake or DBUs in Databricks are just units of compute. One should be able to define what needs to be done and the underlying data warehouse should be capable enough to figure out the size of machine and the parallelism it needs to complete my task within a particular timeframe and minimum cost. No one cares about the configuration of the warehouse if the task succeeds within the business SLA and it’s priced fairly. Though to be honest, accurately predicting the memory and runtime requirements of an SQL before actually executing is a very tough technical problem to solve. Snowflake has always been famous for ‘it just works’ and my gut says that it’ll be the first to solve this problem (if it isn't solved by academia), but for now, it remains unsolved and the users need to take care of this! 

49

13 Comments

🚀 Have you heard of tech-first and tech-enabled companies? Say, Google,Tesla & Walmart,Pfizer respectively - Understanding the distinction is crucial in today’s rapidly evolving business landscape. ⚠️ Caution: The percentages provided are approximations based on my experience in software & data engineering and may vary significantly across different organizations and industries. Share your insights or corrections in the comments. CIO: In tech-first companies (40%), the CIO has strategic oversight, focusing on innovation. In tech-enabled companies (90%), they drive technology adoption and integration for operational efficiency. CTO: Tech-first companies (90%) see the CTO central to tech innovation, while in tech-enabled companies (40%), the focus shifts to integrating technology rather than core development. CDO: In tech-first firms (80%), the CDO innovates data strategies; in tech-enabled companies (60%), they align data strategy with business goals. Data Product Manager: Tech-first (75%) focuses on technology-driven solutions, while tech-enabled (60%) ensures data products meet user needs. Business Analyst: In tech-first companies (60%), they leverage data for new opportunities, while tech-enabled (75%) integrates data into business processes. Data Analyst: Tech-first (70%) emphasizes tech-driven analysis for initiatives, while tech-enabled (80%) focuses on extracting insights for reporting. Pricing Manager: They optimize strategies in tech-enabled (75%) firms and develop dynamic models in tech-first (60%) companies. Data Architect: In tech-first (80%) companies, they create innovative solutions; in tech-enabled (60%), they design data systems. Analytical Translator: Tech-first (75%) firms rely on them to translate insights into strategies, while tech-enabled (65%) firms bridge technical and non-technical teams. Analytics Engineer: They innovate solutions in tech-first (80%) companies, while in tech-enabled (55%), they build analytics infrastructure. Decision Scientist: Their role is more pronounced in tech-first (80%) companies, employing data for strategic decisions, while in tech-enabled (70%), they also use data effectively. Machine Learning Engineer: In tech-first companies (85%), they focus on developing advanced solutions; in tech-enabled firms (50%), they implement models. Applied Data Scientist: Tech-first (85%) firms use them for advanced methodologies; in tech-enabled (60%), they apply algorithms with domain knowledge. Data Scientist: They are deeply integrated in tech-first (90%) companies for product development, while in tech-enabled (10%), they specialize in analysis. Data Engineer: In tech-first (80%), they innovate data systems; in tech-enabled (70%), they maintain data pipelines. Data Governance Officer: They ensure data compliance in both settings (tech-first 75%, tech-enabled 65%). Data Quality Analyst: They focus on data quality in tech-first (80%) and maintain integrity in tech-enabled companies (70%).

27

6 Comments

Databricks just released a new feature called Databricks Apps with the following summary: • Ideal use cases include data visualization, AI applications, self-service analytics, and data quality monitoring. • It supports Dash, Shiny, Grado, Streamlit, and Flask app development frameworks. • Automatic provisioning of serverless compute provides easy app deployment. • Built-in governance with Unity Catalog This is really awesome, but I am struggling keeping up with all of these tools Databricks keeps releasing! We're still trying to figure out the most scalable solution for our automated Workflows and Catalog structure for business consumption, let alone implement the capabilities of the AI/BI Genie, built in AI SQL functions, and now the ability to build a full-blown app 😅 Calm down Databricks! link to article for those interested: https://lnkd.in/gDY_BCgH

1

Want to build a real-time data pipeline with Kafka and Snowflake? ️Blue Orange shows you how! This three-part series dives into building a powerful Change Data Capture (CDC) pipeline to replicate your Postgres databases into Snowflake in real-time. Learn how to: 1. Deploy Confluent Kafka on Kubernetes (even on your local machine!) 2. Automate table replication, schema evolution, and deduplication 3. Turn your data into a high-performance data lake Don't miss this step-by-step guide for building a robust and scalable data pipeline! https://lnkd.in/eTfVUdjC #Kafka #Snowflake #CDC #DataPipeline #Kubernetes #Postgres #RealTimeData #StreamFlake

1