Nick Akincilar’s Post

Data cleansing is getting easier by the day with AI functions built in to modern data cloud products.

Understand step by step 4 Proven Methods to Download Files from Databricks Locally: ✅ Databricks CLI ✅ DBFS Filestore ✅ Databricks REST API ✅ Download results #databricks #dataengineer #dataengineering

🚀 Excited to share insights on the fundamentals of building ETL pipelines with Python and SQL! 💡 In today's data-driven world, the efficiency and scalability of ETL (Extract, Transform, Load) pipelines are paramount. #ETL #Python #SQL #DataEngineering #Modularity #datascience #dataanalytics #AI #ML #AWS

𝗔𝘃𝗼𝗶𝗱 𝗢𝗢𝗠 𝗘𝗿𝗿𝗼𝗿𝘀 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: 𝗧𝗵𝗲 𝗥𝗶𝗴𝗵𝘁 𝗪𝗮𝘆 𝘁𝗼 𝗛𝗮𝗻𝗱𝗹𝗲 𝗗𝗮𝘁𝗮 🚀 If you're working with PySpark, you might have encountered the temptation to use collect() for processing data. But did you know it could lead to Out of Memory (OOM) errors? Here's a quick guide on what not to do—and the right way to handle large datasets. ❌ 𝗧𝗵𝗲 𝗪𝗿𝗼𝗻𝗴 𝗪𝗮𝘆 Using collect() in a loop pulls all the data to the driver node, making it memory-intensive and inefficient: python # Avoid this! def keyValuepair(self): dictionary = {} for row in self.df.collect(): key = row[self.key] value = row[self.value] if key in dictionary: dictionary[key].append(value) else: dictionary[key] = [value] return dictionary Why it’s bad: Data accumulates on the driver, risking OOM errors. Negates Spark's distributed computing benefits. ✅ 𝗧𝗵𝗲 𝗖𝗼𝗿𝗿𝗲𝗰𝘁 𝗪𝗮𝘆 Leverage Spark's distributed processing to avoid memory issues and improve efficiency: python # Use this instead! def keyValuepair(self): return self.df.groupBy(self.key).agg(collect_list(self.value).alias('values')) Why it works: Aggregations are performed on the cluster, reducing driver memory usage. Scalable and efficient for large datasets. Takeaway Only use collect() when dealing with small datasets that fit comfortably in memory. For all other cases, leverage Spark transformations and actions that operate within the cluster. Let’s keep our Spark jobs efficient and scalable! 🚀 Have you faced similar challenges? Let me know your experience in the comments. #PySpark #BigData #DataEngineering #SparkOptimization

Learn Pyspark Step by Step From Scratch 1. What is Pyspark https://lnkd.in/d438wA7v 2. Create DataFrame in Pyspark https://lnkd.in/d_W_Fgc6 3. Read data from CSV Files in Pyspark https://lnkd.in/dCCfzDru 4.Filtering Rows and Columns in Pyspark https://lnkd.in/dSncJm-f 5.Add/Drop Column in Pyspark https://lnkd.in/d2xKekQJ 6.Handling Missing Values in Pyspark https://lnkd.in/dQENr84d

Learn Pyspark Step by Step From Scratch 1. What is Pyspark https://lnkd.in/d438wA7v 2. Create DataFrame in Pyspark https://lnkd.in/d_W_Fgc6 3. Read data from CSV Files in Pyspark https://lnkd.in/dCCfzDru 4.Filtering Rows and Columns in Pyspark https://lnkd.in/dSncJm-f 5.Add/Drop Column in Pyspark https://lnkd.in/d2xKekQJ 6.Handling Missing Values in Pyspark https://lnkd.in/dQENr84d

Learn Pyspark Step by Step From Scratch 1. What is Pyspark https://lnkd.in/d438wA7v 2. Create DataFrame in Pyspark https://lnkd.in/d_W_Fgc6 3. Read data from CSV Files in Pyspark https://lnkd.in/dCCfzDru 4.Filtering Rows and Columns in Pyspark https://lnkd.in/dSncJm-f 5.Add/Drop Column in Pyspark https://lnkd.in/d2xKekQJ 6.Handling Missing Values in Pyspark https://lnkd.in/dQENr84d

Quickly build Analytical Apps on the Lakehouse using "Hudi-rs" It took me 5-10 mins to put a dashboard together in Streamlit. This is just a simple example to show how easily & quickly you can build analytical apps using data directly from open lakehouse platforms like Apache Hudi. All of this is facilitated by "Hudi-rs" - a native Rust library with Python bindings. You can use pure #Python to work with Streamlit & Hudi data without the need to setup Spark, Java & additional dependencies. This means: ❌ Less 'data copies' - usually to cater to such use cases in the data lake, we export a subset of the data from a proprietary warehouse & make copies on copies. ❌ Less data hops (ETL), saving time & leading to fresher insights. ✅ Analysts/Scientists have quicker (& more) access to data for better insights/robust ML models. Usually the wait time on these stakeholders are pretty high. Interested in deep diving? I wrote a detailed blog about 'hudi-rs', its internals & usage last week. Find the link in comments with the code example. #dataengineering #softwareengineering

2. Post: The Key Benefits of PySpark in Big Data Processing?   "PySpark merges the power of Python with Apache Spark’s massive scalability, making it one of the go-to tools for distributed data processing. Whether you’re working on real-time analytics, data warehousing, or building machine learning models, PySpark excels at handling terabytes of data across clusters. Companies like #Yahoo! and #Airbnb use PySpark to process massive datasets while ensuring fault tolerance and scalability. Discover more: [PySpark Overview](https://lnkd.in/gD77sxjn). #BigData #DataEngineering #ML"

❄ 𝗨𝘀𝗲 𝗟𝗟𝗠𝘀 𝗜𝗻 𝗬𝗼𝘂𝗿 𝗦𝗻𝗼𝘄𝗳𝗹𝗮𝗸𝗲 𝗤𝘂𝗲𝗿𝗶𝗲𝘀 🚀 🌟 Snowflake Cortex LLM Functions are now in Public Preview! 🎯 These features are provided as SQL functions and are also available in #Snowpark Python. 🎯 You can easily access #Mistral, #Mixtral, #Llama2 and #Gemma to transform your data with everything running on a single platform--Snowflake! 🎯 CORTEX.COMPLETE(): 👉 Given a prompt, the instruction-following COMPLETE function generates a response using your choice of language model. 👉 In the simplest use case, the prompt is a single string. You may also provide a conversation including multiple prompts and responses for interactive chat-style usage, and in this form of the function you can also specify hyperparameter options to customize the style and size of the output. 🎯 CORTEX.EXTRACT_ANSWER(): 👉 The function extracts an answer to a given question from a text document. 👉 The document may be a plain-English document or a string representation of a semi-structured (JSON) data object. 🎯 CORTEX.SENTIMENT(): 👉 This function returns a sentiment score for the given English-language input text. 🎯 CORTEX.SUMMARIZE(): 👉 This function returns a summary of the given English text. 🎯 CORTEX.TRANSLATE(): 👉 This function translates the given input text from one supported language to another. Article Link - https://lnkd.in/gJtdMs9f 👉Follow Divyansh Saxena for more such content. ------------------------------------------------------------------------- #llm #genai #snowflakedevelopers #snowflakedatacloud  #snowflake #snowflakedb #snowflakesummit #snowflakedeveloper #snowflakedataengineer #snowflake_influencer #snowflakearchitect #awscommunity #awscommunitybuilders #awssolutionsarchitect #awspartner #azurecloud #azure #gcpdataengineer #gcpcloud #dataarchitecture #datacloud #dataandanalytics #databasemanagement #bigdataanalytics #bigdata #bigdatadeveloper #technologyinnovation #engineering #softwareengineers #cloudarchitect #clouddataengineer #ibm #datascience #datasuperhero #learning #data #security #testing #handsonlearning #handsonexperience  #DataEngineeringInterview #InterviewPrep #DataEngineering #TechInterviewTips #data #dataengineering #dataengineerjobs #interviewquestions #interviewprep #newlearnings #interviewpreparation #interviewtips #dataengineers #warehousing #interviewsuccess #DataIntegration #DataArchitecture #dataengineershiring #DataModeling #DataQuality #DataAnalytics #dataloading #bigdataanalytics #bigdata #bigdataengineer #ETL #dataengineerjobs #azurecloud #datacleaning #optimization #interviewsuccess #interviewskills #DataManagement #DataScience #tips #tipoftheday Snowflake Snowflake Developers