Hugo Lu’s Post

🌟 The Future of Data Engineering with Snowflake 🌟 I’m thrilled to share my latest blog: How Snowflake is Shaping the Future of Data Engineering in the Cloud In this post, I dive deep into how Snowflake is transforming the data engineering landscape. From its unique architecture to its scalability and seamless integration capabilities, Snowflake is revolutionizing the way businesses handle and analyze data. Key takeaways: ✅ Understanding Snowflake’s key features ✅ How Snowflake simplifies data management ✅ Real-world applications in modern data workflows If you're exploring Snowflake or data engineering, this post is for you! Let’s discuss how Snowflake is changing the game in the comments. 🔍💬 #DataEngineering #Snowflake #CloudData #DataWarehouse #BigData #Python #ControlM

Loome makes building and maintaining a modern #DataWarehouse frictionless. With over a hundred pre-built connectors including #AzureSynapse, Databricks and Snowflake you can set up and have full visibility over your data pipeline. The ability to orchestrate multiple process types including #SQL, #PowerShell, Command Line, #Databricks, #Python and more provides you the flexibility to leverage processes you already have in your existing data workflows. Find out more. #CloudComputing #ModernDataPlatform #DataPipeline #DataLakes 👉 https://bit.ly/4c8U8wh

🚀 Exploring Data Engineering with Snowflake and Python! 🚀 Recently, I participated in a Snowflake workshop where I had the chance to work hands-on with Snowflake’s Snowpark and pandas. 📌 Workshop Highlights: - Data Aggregation and Visualization: Used Snowpark pandas to transform and visualize data directly within Snowflake. - Feature Engineering: Created new features and stored the results efficiently, leveraging Snowflake's robust processing power. - Orchestration: Set up serverless tasks to automate and schedule parts of the workflow, with Git integration to manage version control. Check out the notebook here: https://lnkd.in/gTCYhuD5 #DataEngineering #Snowflake #Python #Pandas #BigData #LearningJourney

Hi, How we upload the Json file and multiple Json file in Pyspark with databricks. Thank You 🙂 Below given the link 👇 https://lnkd.in/gem9R9eP #connection #connect, #databricks, #python

Hello, Connections! 𝐃𝐚𝐭𝐚𝐛𝐫𝐢𝐜𝐤𝐬 𝐦𝐚𝐠𝐢𝐜 𝐜𝐨𝐦𝐦𝐚𝐧𝐝𝐬 ✨ Databricks magic commands are special commands that allow you to interact with Databricks notebooks more efficiently. They are prefixed with a % symbol. ➡ %𝐩𝐲𝐭𝐡𝐨𝐧, %𝐬𝐪𝐥 , %𝐬𝐜𝐚𝐥𝐚 ,%𝐫 Use these commands to write code for different programming languages in a notebook cell. ➡ %𝐣𝐨𝐛𝐬:- lists all the running jobs. ➡ %𝐦𝐝 (𝐌𝐚𝐫𝐤𝐝𝐨𝐰𝐧):- This magic command is used to render markdown. ➡ %𝐬𝐡:- Run shell commands within a notebook. ➡ %𝐟𝐬:- This command interacts with Databricks' file system (DBFS). ➡ %𝐫𝐮𝐧:- Executes another notebook from the current notebook. ➡ %𝐝𝐛𝐮𝐭𝐢𝐥𝐬:- Use this to run utility functions (e.g., managing file systems, secrets, etc.) ➡ %𝐡𝐢𝐬𝐭𝐨𝐫𝐲:- Shows the history of commands run in the notebook. You can specify ranges or search terms to narrow down the output. These magic commands help streamline workflows in Databricks by allowing different languages, file management, and other tasks within a notebook environment. 👉 𝐉𝐨𝐢𝐧 𝐦𝐲 𝐓𝐞𝐥𝐞𝐠𝐫𝐚𝐦 𝐂𝐡𝐚𝐧𝐧𝐞𝐥 𝐭𝐨 𝐥𝐞𝐚𝐫𝐧 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 & 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬: https://t.me/LK_Data_world 👉 If you found this post informative, save and repost it🔁. 👉 Lovee Kumar 🛎 #data #dataengineer #bigdataengineer #datascience #spark #bigdata #databricks #skills #guide #pyspark #learning #linkedinconnection

Have you ever wondered how we can extract insights from complex sports data? 🤔 This serverless AWS ETL pipeline shows you how: Data Lake → Lambda Function → HTTP API Gateway → Insights and Plots. Check out full project: [https://lnkd.in/eMt4ggcr] #DataEngineering #AWSLambda #DataAnalytics #Python #AWS #UFC

Hello, Connections! 𝐃𝐚𝐭𝐚𝐛𝐫𝐢𝐜𝐤𝐬 𝐦𝐚𝐠𝐢𝐜 𝐜𝐨𝐦𝐦𝐚𝐧𝐝𝐬 ✨ Databricks magic commands are special commands that allow you to interact with Databricks notebooks more efficiently. They are prefixed with a % symbol. ➡ %𝐩𝐲𝐭𝐡𝐨𝐧, %𝐬𝐪𝐥 , %𝐬𝐜𝐚𝐥𝐚 ,%𝐫 Use these commands to write code for different programming languages in a notebook cell. ➡ %𝐣𝐨𝐛𝐬:- lists all the running jobs. ➡ %𝐦𝐝 (𝐌𝐚𝐫𝐤𝐝𝐨𝐰𝐧):- This magic command is used to render markdown. ➡ %𝐬𝐡:- Run shell commands within a notebook. ➡ %𝐟𝐬:- This command interacts with Databricks' file system (DBFS). ➡ %𝐫𝐮𝐧:- Executes another notebook from the current notebook. ➡ %𝐝𝐛𝐮𝐭𝐢𝐥𝐬:- Use this to run utility functions (e.g., managing file systems, secrets, etc.) ➡ %𝐡𝐢𝐬𝐭𝐨𝐫𝐲:- Shows the history of commands run in the notebook. You can specify ranges or search terms to narrow down the output. These magic commands help streamline workflows in Databricks by allowing different languages, file management, and other tasks within a notebook environment. 👉 𝐉𝐨𝐢𝐧 𝐦𝐲 𝐓𝐞𝐥𝐞𝐠𝐫𝐚𝐦 𝐂𝐡𝐚𝐧𝐧𝐞𝐥 𝐭𝐨 𝐥𝐞𝐚𝐫𝐧 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 & 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬: https://t.me/LK_Data_world 👉 If you found this post informative, save and repost it🔁. 👉 follow for more Lovee Kumar 🛎 #data #dataengineer #bigdataengineer #datascience #spark #bigdata #databricks #skills #guide #pyspark #learning #linkedinconnection

Tired of spending Snowflake / BigQuery / DX credits to just check and load raw data into a querable format? What if I tell you those workloads can be moved *before* the warehouse without writing a single line of #Java? We open-sourced a reference implementation of Write-Audit-Publish with #PyIceberg (from Tabular) and #PyNessie (from Dremio) on serverless compute. The cherry on top? We show how the final, post-ETL table can be queried with #snowflake, thanks to the #Iceberg format. The Python data ecosystem is in its early days (Fokko Driesprong, Alex Merced: when an official integration? 😎), but times are a-changin': want to start on a full Pythonic, no-nonsense lakehouse and don't know how? Ping us anytime. Repo by yours truly and Luca Bigon: https://lnkd.in/eB4yTWzw Post by Ciro Greco: https://lnkd.in/eGMVKj6p #dataengineering #lakehouse #opensource #s3 #dataquality #serverless #python

Currently I am getting some hands-on experience with the just announced LocalStack for Snowflake emulator. The Snowflake emulator supports the following features: 🛠 Basic operations on warehouses, databases, schemas, and tables 🗄 Storing files in user/data/named stages 📚 Snowpark libraries 🎞 Snowpipe streaming with Kafka connector 📜 JavaScript and Python UDFs … and more! And it can also be integrated into a CI/CD pipeline. Once I am done testing I'll let you know about my experiences here on LinkedIn :D. Looking forward to this ❄❤️. #data #dataengineering #testing

🧱 Databricks data engineers, did you know you can import your own Python modules and just use your notebook as a wrapper/orchestrator? If you don't have a private PyPi, this is the best hybrid model between running wheels and notebooks. I use it all the time! For everyone that has productionalized Python before, this will be a walk in the park. It will allow you to import your own custom Python modules inside your notebooks, i.e. 𝒇𝙧𝒐𝙢 𝙢𝒚_𝒍𝙞𝒃𝙧𝒂𝙧𝒚.𝒎𝙮_𝙢𝒐𝙙𝒖𝙡𝒆 𝒊𝙢𝒑𝙤𝒓𝙩 𝙢𝒚_𝒇𝙪𝒏𝙘𝒕𝙞𝒐𝙣, 𝑴𝙮𝑪𝙡𝒂𝙨𝒔 So how does it work? This will be a bit too long for a LinkedIn post, so I have put the entire example up on my Github page: https://lnkd.in/emDap8kU. Or you can look at the below image and have a basic idea! So what are some of the pros of doing this? - Ability to use precommit hooks to do linting and PEP8 checks - Follow software engineering best practices - Better IDE experience (although Databricks notebooks are getting there) - Ability to include and run tests much more easily - Increase modularity by organizing your code in seperate files - Reusability across workloads/perhaps even workspaces - Ability to use Databricks' first class citizens (notebooks) while being fully in control of the code it runs - Your notebooks will be really really clean! 🧼 - 𝗣𝗿𝗼-𝘁𝗶𝗽: 𝗜𝘁 𝘄𝗼𝗿𝗸𝘀 𝗿𝗲𝗮𝗹𝗹𝘆 𝗿𝗲𝗮𝗹𝗹𝘆 𝘄𝗲𝗹𝗹 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗔𝘀𝘀𝗲𝘁 𝗕𝘂𝗻𝗱𝗹𝗲𝘀! There are some cons too: - To make your local env happy, you need to have a custom DBUtils function - For all Spark workloads, you need to take over Databricks' SparkSession Solutions for these cons are in the repo too. For me, the pros outweigh the cons! #dataengineering