Rajat Jain

If you are interested in integrating dbt transformations with PostgreSQL directly within Airflow using the Cosmos package then this post is for you. With Cosmos, profile mapping is a breeze, making it easier than ever to leverage dbt in orchestrated pipelines. This setup streamlines data transformation tasks by managing dbt dependencies and profiles through Airflow. If you're looking to make your dbt workflows even more efficient, the Cosmos package is worth exploring! 👉 Link is in the first comment #dbt #PostgreSQL #Airflow #DataEngineering #Cosmos

25

3 Kommentare

🌟 Exciting News for Snowflake Enthusiasts in Hyderabad! 🌟 The #Snowflake Hyderabad User Group is back with another amazing in-person meetup, hosted by Snowflake. Join us on December 07, 2024, from 9:00 AM to 1:00 PM, for a hands-on workshop led by Satish T, Cloud Data Architect at Infocepts Pvt Ltd and Snowflake Data SuperHero. In this workshop, you’ll learn how to: 🔍 Translate multilingual customer reviews 📊 Categorize and analyze unstructured review data at scale ⭐ Extract ratings and sentiments from reviews 💡 Identify customer issues and take action with LLM assistance 💻 What You’ll Build: We’ll analyze Tasty Bytes customer reviews using Snowflake Cortex within Snowflake Notebooks to discover: ✅ How likely customers are to recommend Tasty Bytes ✅ Sentiments around food, pricing, and other aspects ✅ Key issues customers highlight and how to address them 🎯 Prerequisites: Familiarity with Python and DataFrame API Experience with Snowflake and Snowpark A Snowflake account with access to Cortex LLM functions/models and Snowflake Notebooks. Don’t miss this opportunity to level up your skills with Snowflake Cortex. #Snowflake #LLM #dataengineer #Cortex #dbt #genai

12

Hi All, I am curious on Databricks Serverless. There are three types of scenarios I see: 1. New Implementations on Databricks serverless compute. 2. Existing implementation on Databricks classic compute planning to shift to Databricks Serverless compute. 3. Teams working on competitor offering in serverless such as EMR serverless planning to evaluate Databricks Serverless compute. Problem: Except for a blog there is no concrete comparison or document available on Databricks serverless compute launched in June 2024. Any help on this topic or documents and links is appreciated? Please do comment and share. #Databricks #DataServerless

20

3 Kommentare

🚀 Excited to share my article on deploying Large Language Models (LLMs) as custom endpoints from Databricks Marketplace! 💡 Discover how these LLMs can be seamlessly integrated into Databricks notebooks and various applications like Retrieval Augmented Generation (RAG). Let's unlock the potential of AI together! #AI #Databricks #LLMs #OpenSource #Notebooks #RAG #Innovation #DataScience #MachineLearning

32

1 Kommentar

Before diving into the world of Databricks, take a step back and master the fundamentals of Spark. Spend good time to understand the Spark Core Architecture & Fundamentals. You should have fair understanding "How Spark works"?? Understand the Roles of Driver Workers Executors Task Cluster Manager Spark Context RDD Action Transformation DAG / Task Schedulers By understanding these core concepts, you'll have a solid foundation for building efficient and scalable Spark applications. #spark #databricks #upskill #dataengineering #learning

107

4 Kommentare

Feedback like this encourage us to put more efforts. Snowflake is one of the most demanding skill for data engineers these days. If you are coming from Pl/SQL or ETL backgrounds, you can make a smooth transition. #snowflake #feedback #upskill #training #feedback

65

3 Kommentare

Ever wondered about Data Factory deployments using Infrastructure as Code (IaC) and where to draw the line on where to stop coding infrastructure inside your Factory? You shouldn’t build pipelines for example using IaC, rather it should be part of development code, not infrastructure code. See my blog about it here https://lnkd.in/ejnGDUji

4

SPARK code to impute NULL or missing values in a table containing NULL in multiple columns. #SPARK #SCALA #BIGDATA Karthik K. def main(args:Array[String]):Unit = { val ss = SparkSession.builder.master("local[*]").appName("DFDEMO").getOrCreate() import ss.implicits._ val df_1 = ss.read.format("csv").option("header",true). option("path","C:/Users/hp/COMPANY.txt").load() df_1.show() val df_2 = df_1.select(col("PRODUCT_NAME"),col("PRODUCT_ID") , (when(col("PRODUCT_ID").isNull,0).otherwise(col("PRODUCT_ID")).as("LEGIT_PRODUCT_ID")), when(col("PRODUCT_NAME").isNull,"UNKNOWN COMPANY"). otherwise(col("PRODUCT_NAME")).as("LEGIT_PRODUCT_NAME")) df_2.show() OUTPUT:

26

4 Kommentare

🚀 Understanding Apache Spark Architecture: Key Components, RDDs, and Their Role in Big Data Processing 🚀 Apache Spark has transformed the landscape of big data processing with its unified platform for batch and stream processing. In my latest Medium article, I delve into the architectural brilliance that makes Spark a go-to tool for data engineers and scientists. 🔑 Key Highlights: 1) Resilient Distributed Datasets (RDDs): Learn why RDDs are the backbone of Spark's fault tolerance and parallel processing capabilities. 2) Core Components: Explore the roles of the Driver, SparkContext, Cluster Manager, Executors, and Tasks in Spark’s architecture. 3) RDD Transformations and Actions: Understand the importance of transformations and actions in Spark's lazy evaluation model. 4) Fault Tolerance: Discover how Spark ensures data consistency and fault tolerance through RDD immutability and DAG lineage. Curious about how Spark stacks up against Apache Flink? I’ve got you covered with a comparative analysis on fault tolerance, memory management, and scalability. Whether you’re a data enthusiast or a seasoned professional, this article provides valuable insights into leveraging Spark for your big data needs. 📖 Read the full article here: [https://lnkd.in/gDujQt-z] #BigData #ApacheSpark #DataEngineering #DataScience #RDD #SparkArchitecture #TechInsights

5

With Large Data Volumes and Aggressive SLAs, building low latency Cost Optimum, Declarative and continuous Data Pipelines is need of the hour. Here's my latest article on Snowflake Dynamic Tables. #delcarativepipeline #continouspipeline #dynamicrefresh Please feel free to like, Subscribe and Share my article and Newsletter.

3

Databricks Solutions Architect Essentials !!! This one is very unique. 3 days of intense databricks workshop and an assesment to validate the skills. It is currenly available only for databricks partners. My heartfelt thanks to Sushil Nithiyanantham & Sony Sankar for the guidance and support. Many thanks to Tiger Analytics for the guidance and support. #tigertribe #databricks #databrickssolutionsArchitect.

96

4 Kommentare

Future demands more Cloud Data Engineers; Reliance increasing footprints in Cloud Mukesh Ambani in its 47th AGM launches Jio AI Cloud wherein Jio users will get 100 GB free cloud storage starting this Diwali. This is just start but we all have seen the speed of Reliance in Jio. This AI Cloud will move more fast and will make Indian small business to start going for cloud services. It can be the game changer for Indian economy and also in Cloud industry. The demand & future of data engineers can be now judged by big players like Reliance building businesses around this domain. Knowing Cloud with Big Data is the perfect profile which industry look for today. Google recently partnered with Adani to supply clean energy for Google cloud operations. Remember Data Engineers act as backbone for AI, providing structured data in usable form. The future is of Data & Data Engineers are going to rule this domain for sure. I have prepared 1000s of Data Engineers in last 5 years, who work at Top product-based companies today. My Intensive Cloud focused Big Data Program has been created to make Top Data Engineers, who can work on complex projects in industry. New batch starting today.

1.782

44 Kommentare

Is Your Data Architecture Stuck in the Past? The Pitfalls of Non-Lakehouse Solutions !! Lack of Unified Governance Trade-off Between Performance and Flexibility Data Silos for managing data lakes and data warehouses separately Lakehouse architecture blends the strengths of data lakes and warehouses, addressing above concerns. It offers a unified platform that resolves common issues in data management, providing scalability, reliability, and flexibility. I recently delved into this innovative approach through the Databricks Fundamentals training. #DataLakehouse #DataArchitecture #BigData

10

1 Kommentar

My blog on "Migrate #data from an on-premises #Hadoop environment to #Amazon #S3 using #S3DistCp with AWS Direct Connect" is published on #AWS Blogs https://lnkd.in/eqVEFEcV #dataengineering

11

Noticed that the Ambarish Ganguly Academy Azure AI Search videos have been appreciated by the viewers. Providing one of the videos which has got a lot of views and good feedback RAG with Azure AI Search with Azure OpenAI with different search techniques and Langchain with Conversation Chain, Prompt Template and Conversation Buffer   📍  Vector Search   📍  Hybrid Search     📍  Exhaustive KNN Search     📍  Hybrid Search with Semantic reranking   📍  RAG with Azure AI Search with Azure OpenAI     📍  Use Langchain with Conversation Chain       📍  Use Langchain with Prompt Template   📍  Use Langchain with Conversation Buffer     Video: https://lnkd.in/gy3GDr9e Code: https://lnkd.in/gcVY-SSg #rag #azureopenai #vectorsearch #hybridsearch #semanticsearch #llm #azureaisearch #azureai

45

5 Kommentare

💡 Many, many years ago when I read Tom Kyte's Expert One on One Oracle cover to cover, I understood many concepts of the RDBMS very thoroughly and they have stood me in good stead over the years. 👉 One of the concepts the book explained in some detail was how logical input output (LIO), which translates as a read from memory, is orders of magnitude faster than physical input output, which is a read from the disk. 🔦 Over the years, all relational databases did everything to ensure you were doing a read from memory rather than a read from the disk (plain speak!). So, for that you would have an ever increasing need for memory, because ideally you want to have as many data blocks as possible in your buffer cache--so that kind of begs the question, why not have all the data blocks in the cache? We can, and that is the genesis of in-memory databases. However in-memory databases are not manna from heaven, they have their own woes. I am not going into that here. What I would like to touch upon is having a traditional database benefiting from the features of an in-memory database. 👉 In this context, I would like to mention Amazon Aurora database's Optimized Reads, where you can now take advantage of the locally attached NVMe solid state drives (SSD) available on db.r6gd and db.r6id instances. The Optimized Reads tiered cache enhances DB instance caching by integrating local NVMe storage directly into the database buffer pool. This local caching on NVMe storage results in quicker response times compared to using Aurora network storage. 👉 As a result, Optimized Reads boosts price-performance efficiency by minimizing the need for excessive instance provisioning solely to decrease query latency. Additionally, it offers greater flexibility to extend your application's working dataset beyond the memory capacity of the largest instances, allowing you to derive more rapid and comprehensive insights from your data. #aurora #aws #databases #performancetuning Prashant Dixit Uma Vedula Yash Bindlish Ashish Jha Parna Mehta Ashish P. Manoj Kale Disha Babla Sunil Patro Shashank Chinchli Shri Hamilpurkar Deepika Nair Suraj Dutt Santapan Sarkar Jayashree Ramesh Debjani sen Somraj Chakrabarty Surajit Roy Ajay Gulati ARGHYA SEN Samreen Taj Arup Das

26

3 Kommentare

#GoogleCloud #BigQuery #DataProc #CloudComputing #Pandas #ETL #ELT #DataAnalytics I tried to put some thoughts in different options of data processing in GCP cloud platform with Dataproc, BigQuery and Pandas. It is a comparative analysis one these three options based on performance and cost. There is no best or worst solution here, it all depends on your need and business requirements. Feel free to read through this short article, do put your thoughts 😊 Hope you enjoyed 😊 😊 😊 😊

7

Latest blog on how to connect Oracle Autonomous Data Warehouse (ADW) to Apache Superset using the python-oracledb thick client on an Oracle Cloud Infrastructure (OCI) compute instance. Connection between Oracle ADW and Apache Superset can be quite the maze, given the scattered pieces of information out there. This step-by-step guide brings all the necessary details together in one place, making it much easier for you to set up and start exploring your data without much hassle. Thanks to @christopher jones (https://github.com/cjbj). His insights have been helpful for this. https://lnkd.in/g6h5Fi5u #Superset #PythonOracledb #OracleCloud

10