Dhilip Subramanian’s Post

Experimented with #snowpark for Snowflake by Vino Duraisamy for running my data engineering pipeline using python and found it very intuitive and seamless offering key learnings: - Learned about snowflake client side libraries (particularly, Dataframe APIs) - Server-side UDFs. - Aggregate and join functions useful for data processing. try it out.

It seems like Snowflake has released a preview of notebooks. But for some reason I cannot find anything related to this topic in the documentation. So far it seems like it is easy to integrate your Git-repo and it supports SQL, Python and Markdown. It will be fun learning more about this new little gem. #snowflakedb #snowflake

Databricks Notebook Debugger! Here's how to leverage this feature with a small PySpark code. Requirements: 1- Enable the debugger feature from the workspace developer settings 2- Use a cluster with Databricks Runtime version 13.3 LTS or above. Thanks to Derar Alhussein Hubert Dudek Denny Lee for sharing unique features. Code in the comment section. Official Doc: https://lnkd.in/dfc25e8u #databricks #sql #python #dataengineer #sparksql #etl

🚀 Excited to Share My First Tech Blog! 🚀 After months of hands-on experience and analysis, I've published my first Medium article, titled "Build your own CI/CD Pipeline for Snowflake, Airflow, DBT, Python & Other Configuration Files (Data Pipeline)". 🎉 This blog is a comprehensive guide for data engineers, covering the design and implementation of an end-to-end CI/CD pipeline. Whether you're dealing with Snowflake, Airflow, DBT, or other configuration files, this article is designed to help you streamline deployments efficiently and effectively. 🔗 Check it out here: https://lnkd.in/gh7JGhtp I'm eager to hear your feedback and would love for this to spark insightful discussions. If you find it useful, please like, comment, or share it with your network. Your support means a lot as I take this exciting step in sharing knowledge with the community! #DataEngineering #CICD #MediumBlog #Snowflake #Airflow #DBT #Python

🧠 Have you been wanting a Snowflake-native interface to - Write Python & SQL for data analysis - Perform feature engineering and run #ML models - Automate #dataengineering pipelines ... and more, with all of your code checked into #Git❓ Come chat with us on June 4! At Snowflake #Summit, Tyler Simons and I will talk about how you can do all of the above in the brand new 🌟 Snowflake Notebooks 📓!

Empowering data engineers! Do check out this comprehensive guide to building CI/CD pipelines for Snowflake, Airflow, DBT & Python by my amazing team member Sanjay R B. #DataEngineering #CICD"

Discover the future of data workflows with Snowflake Notebooks! Seamlessly integrate Python, SQL, and Markdown for enhanced productivity and collaboration. Unlock powerful features like Snowpark ML, Streamlit, native Git support, and advanced ML workflows. Transform your data processes today! #cci #cloudconsultings #snowflake #DataScience #MachineLearning #Analytics #SnowflakeNotebooks

I've been experimenting with Rust for data engineering and, specifically for single-node applications. There is an ecosystem (and community!) brewing and thriving that is waiting to explode. See Polars (with bindings for Python); as a direct replacement for Pandas, the umbrella project Apache DataFusion (which includes Comet and Ballista) competes with Apache Spark. I'll be dropping a few deep dives and snippets here and there ;) #rust #dataengineering #datafusion #apachespark

I’m happy to share that I’ve obtained a new certification: Hands-On Essentials: Data Application Builders Workshop from Snowflake! Take aways: Almost like a full-stack developer bootcamp, this course covers a wide array of technologies that will help you build applications that use Snowflake as a back end. Including: Streamlit (Python), SnowSQL, REST APIs, and much more. #Snowflake #Streamlit #API

💡 Generic Magic Command for Running Notebooks: To run a notebook from another notebook, use %run notebook_path. 🛠️ Databricks Utilities (dbutils): Databricks notebooks offer utility commands like dbutils for environment interaction. Use dbutils.fs.ls() to list directories of files from Python cells, like: display(dbutils.fs.ls("/databricks-datasets")) 📊 Displaying Tabular Data: Use the display() function when you have tabular data returned by a Python cell. #Databricks #Notebook #Utilities #DataVisualization 📊