Abhinandan Patil

Thrilled to achieve the Microsoft Certified: Fabric Analytics Engineer Associate certification! 🙌 Heartfelt thanks to my COE team, Tiger Analytics, and amazing colleagues for their unwavering support. Excited to apply my skills and drive impactful analytics solutions! 🚀 #MicrosoftCertification #FabricAnalyticsEngineer #ProfessionalGrowth #TeamSupport #AchievementUnlocked #ContinuousLearning #CareerMilestone #DataAnalytics #DataEngineering #TigerTribe #TigerAnalytics

31

6 Comments

🚀 Had an enriching experience at the Data Intelligence Day event by Databricks! Here's a deeper dive into what I learned: Insights from Databricks Leaders: Anil Bhasin, Ed Lenta, and Saroj V. provided invaluable insights into the Databricks platform. They highlighted its capabilities in data governance, warehousing, and GenAI. The platform's seamless integration across clouds and platforms stood out, offering organizations a unified solution for their data and AI needs. DBRX LLM: The discussion on DBRX, an open general-purpose LLM, was eye-opening. It was fascinating to learn about its potential to other established models like GPT, LLaMA, and Mistral. This signifies a significant leap forward in democratizing AI and making advanced AI capabilities more accessible to a wider audience. GenAI Impact in Banking and Multilingualism: - Rana Sinha Ray from HDFC Bank shared compelling use cases of GenAI in the banking sector. The potential for AI to revolutionize processes and customer experiences was evident. - Gautam Bhargava's presentation on Krutrim's multilingual GenAI model showcased the power of AI in addressing linguistic diversity. This innovation has immense implications, especially in a diverse country like India. Panel Discussion on Industry Trends: The panel discussion, moderated by Shweta Kamath, brought together industry leaders like Kamal K., Atul Sawant, Chintan Shah and Sudip Mazumder. Their insights on current trends and challenges in data intelligence and AI were enlightening. The discussion underscored the importance of continuous innovation and collaboration in driving industry growth. Key takeaways for me: - Krutrim's launch of a multi-lingual model is revolutionary and has the potential to make a significant impact in a diverse country like India. - Databricks's Unity Catalog emerged as a powerful tool for streamlining data governance and accelerating AI initiatives. - The increasing integration of GenAI into everyday life highlights the growing significance of data intelligence and AI technologies. Lastly a big thank you to Sathwik HV and Alankrit Pareek for inviting Fynd (Shopsense Retail Technologies Ltd.) team and managing registration process seamlessly. Your efforts ensured a smooth and enjoyable experience for all attendees! #DataIntelligenceDay #Databricks #AI #Innovation 📈✨

40

TGIF, folks! Fridays are hands down my favorite day—a perfect mix of weekend anticipation and a well-deserved breather from the weekly hustle (which I also love, haha). So here's the latest, I've finally caved and set up a Topmate profile! 😅 Why? Because so many of you folks have been reaching out for 1-1 help with: 1. Navigating the wild world of data science and analytics careers 2. Prepping for those nerve-wracking interviews I’ve decided to dedicate few hours of my weekend to offering personalized support for these requests. I've opened up a few 1-on-1 slots on my Topmate profile for those interested. If you're looking for guidance or interview prep, feel free to check it out and book a session! Looking forward to helping you on your data science journey. 😊 https://lnkd.in/gy-fMC6Q #CareerGuidance #DataScience #Analytics #InterviewPrep #TopmateProfile

38

1 Comment

LakeFlow + Delta Live Tables + Serverless Autoscaling Compute sound like a perfect solution to load all required data with data expectations / monitoring / alerting to the Lakehouse :)

12

It has been fun putting together the Data Engineering track for Fifth Elephant Conference this year along with Shashwat Agarwal and Pramod Biligiri #DataManagement, #MachineLearning, #AI & #Data We have assembled a diverse set of talks.   Even if you are not a hard-core data engineer, you will find something of interest A few talks to highlight 1. Vishnu Vasanth from e6data will talk on the Lakehouse Compute engine they have built. 2. Ruturaj Bhokre from Dream11 will talk on the Evolution of their data platform 3. Mohd. Shadab S. (CISO at Hotstar) will explain Design patterns in Data security 4. Niraj Mishra from Atlassian will talk on Ephemeral data pipelines You may know of API contracts but have you heard of Data Contracts ? Bichitra Kumar Sahoo, Rahul Madan and Shikhar Vaish from Atlan will be conducting a workshop on Data contracts and OpenLineage. There is a separate "LLM and GenAI" track as well, if that suits your interest more. The Fifth Elephant 2024 Conference is on: 📅 13 July 2024 (Saturday)  ⏰ 9 AM to 6 PM  📌 BIC, Indiranagar, Bangalore + Hybrid Access Check out the schedule here & go grab your early elephant tickets today at: http://has.gy/Xz_w P.S. Tag anyone in your networks who is working in the space of #AI #DataScience & #LLMs

29

🚀 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗶𝗻𝗴 𝗗𝗮𝘁𝗮 𝗪𝗮𝗿𝗲𝗵𝗼𝘂𝘀𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗗𝗲𝗹𝘁𝗮 𝗟𝗮𝗸𝗲 A logistics company managing petabytes of data modernized their legacy warehouse with Databricks Delta Lake, addressing challenges like real-time pipelines, data-sharing, and scalability. Challenges Addressed • Real-time batch & streaming pipelines with low latency. • Petabyte-scale data management & secure collaboration. • CDC workflows & cross-platform integration (Iceberg). Delta Lake in Action • Efficient Storage: Delta tables (Parquet) optimized petabyte-scale data. • Liquid Clustering: Enhanced query speeds by reorganizing frequently updated data. • Delta Sharing: Enabled secure, real-time data collaboration across stakeholders. • Delta Live Tables: Simplified multi-hop pipelines (Bronze, Silver, Gold). • Performance Optimization: Z-Order compaction and vacuuming for lean storage. • Unified Analytics: Real-time insights with Databricks and MLflow. Outcomes 🌟 Optimized Queries: Liquid Clustering + Z-Order ensured efficiency. 🚀 Seamless Collaboration: Delta Sharing enabled governed external sharing. 📊 Real-Time Analytics: Unified batch & streaming unlocked actionable insights. 🛠 Efficient Resource Usage: Retention management kept storage lean. 💡 Takeaway: Delta Lake’s features like Liquid Clustering and Delta Sharing empower scalable, real-time, and governed data solutions for modern enterprises. How is your team leveraging lakehouses? Let’s explore your journey! 🚀 Stay connected and follow Abhijit Das on LinkedIn for insights and updates #DataEngineering #DeltaLake #RealTimeAnalytics #LiquidClustering #DeltaSharing #DataLakehouse #Databricks

7

🚀 Exciting day at Snowflake Data Cloud World Tour Mumbai 2024! ❄️ I had the privilege of attending the Snowflake Data Cloud World Tour in Mumbai, and here are some key takeaways from insightful sessions on the future of data and AI: 📊 Open Table Formats & Iceberg Support Exploring integration possibilities with Apache Iceberg and Polaris Discussions on ACID compliance, performance optimization, and data evolution 🛡️ Snowflake Horizon Unveiling a vision for unified data governance and security across clouds Concepts for simplifying management of data and apps from a centralized interface 🤖 Generative AI & LLMs Introduction to the Snowflake Arctic family of Large Language Models (LLMs) Exploring Snowflake Cortex as a pathway to democratize AI Emphasis on the "AI to Data, not Data to AI" approach for enterprises 🔒 Data Governance & Security Highlighting the importance of unified security and compliance measures Showcasing potential applications of Dynamic Data Masking for data protection 💻 Snowpark Container Service & ML API Demonstrations of running AI and full-stack apps within the Snowflake ecosystem Exploring possibilities for direct ML model execution and predictions 🔗 Native Apps & Data Sharing Discussing streamlined approaches to building and sharing data applications Insights into enhancing operational collaborations through data sharing 📄 Document AI & Unstructured Data Previewing tools aimed at extracting insights from unstructured data The event showcased Snowflake's vision for innovation, open standards, and efficient data management. It was fascinating to see the proposed solutions for various data challenges, from governance to AI implementation. Kudos to the speakers, including Sanjay Deshmukh Vijayant Rai Rahul Jain 🇮🇳 Kamesh Sampath and Sumeet Tandure Excited to contemplate how these concepts might shape our future data strategies! #SnowflakeWorldTour #DataCloud #AI #DataGovernance #Innovation

51

2 Comments

I am excited to share a recent project where I explored Apache Airflow to orchestrate data workflows, integrating AWS S3, Snowflake, and DBT to build an automated data pipeline. As a data engineer, this project has given me valuable hands-on experience in workflow orchestration, automating tasks, and managing dependencies to ensure efficient data processing. 🚀 Key Workflow - S3 to Snowflake ETL - Developed the DAGs that monitors S3 for new data, processes it, and loads it into Snowflake, utilizing operators, hooks, and sensors. Dynamic Task Grouping & XCom - Organized tasks using TaskGroup and passed data between tasks using XComs. Branching Logic - Implemented conditional branching based on file availability in S3, sending alerts or continuing with the ETL process as necessary. Email Notifications - Set up SMTP for automatic email alerts on job success or failure Tech Stack - Apache Airflow, AWS S3, Snowflake, DBT, Docker It’s been a great learning journey, and I look forward to building even more advanced and efficient data solutions moving forward. Feel free to connect if you’re working on similar projects or want to discuss about Airflow and cloud-based data pipelines! Your feedback is always appreciated! 🙌 #DataEngineering #ApacheAirflow #AWS #Snowflake #DBT #DataPipelines #ETL #Cloud #Automation #DataEngineeringProjects

6

2 Comments

I'm excited to announce the release of Databricks Assistant Quick Fix. Upon cell errors, Databricks Assistant will automatically provide single-line suggestions for simple SQL and Python errors we have high confidence can be resolved by an LLM. This includes errors like: trailing columns, misspelled table and column names, syntax errors and more! We designed Quick Fix to be fast! Suggestions should take 1-3 seconds and we've seamlessly integrated it with keyboard shortcuts. Try it out today and don't hesitate to reach out with your feedback— we'd love to hear your thoughts! #DatabricksAssistant #QuickFix #AI #LLM #DataScience #SQL #Python #Databricks #Assistant #Notebooks

76

6 Comments

Just submitted my creation, Arctic Analytics (https://lnkd.in/gVehD56b) to The Future of AI is Open hackathon conducted by #Snowflake and #Streamlit, and what a ride it's been! 🏁💥 Arctic Analytics a Streamlit app based on Snowflake's Arctic AI model, represents a leap toward bridging data privacy and analytics accessibility. It's not merely a tool but a statement - ethical AI use can empower anyone to unleash their data's potential, all the while maintaining stringent data privacy. Imagine drawing powerful insights and visuals without the AI ever peeking into your data, coupled with unmatched transparency about the data shared with the AI. 🛡️🤖 I am proud to have navigated this venture, merging technology with values. And this is merely the start. As submissions close and I await results on June 6, Arctic Analytics stands as a testament to what we can achieve at the intersection of privacy, accessibility, and innovative technology. 🔥🌐 Ready to usher in the future of AI with open arms and secure data? Join me in pushing tomorrow's boundaries today! 🌈⏳ #Hackathon #AI #Innovation #DataPrivacy #TheFutureOfAIIsOpen #SnowflakeArctic #ArcticAnalytics

73

2 Comments

I’m excited to share my latest blog, "Transforming Insights into Solutions: My Takeaways from Snowflake SPN Connect". In the blog I have shared my learnings from the Snowflake SPN Connect event, exploring powerful new features like Data Quality Monitoring, Snowflake Analyst, and GIT Integrator, along with some innovative ways to leverage them for smarter data solutions. I’d love to hear your thoughts and feedback! Let’s discuss how these technologies can be applied to real-world problems.

31

1 Comment

A lot of people try deep learning for financial forecasting, only to realize time-series models often win. If you are passionate about techifying finance, check out Irina Zhecheva's post. Here’s why her team is awesome: • 𝐈𝐦𝐩𝐚𝐜𝐭 ✅: Irina and her team have spent 7+ years building automated forecasting data products that execs 𝘢𝘤𝘵𝘶𝘢𝘭𝘭𝘺 use. • 𝐓𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐬𝐤𝐢𝐥𝐥𝐬 ✅: Before Amazon, Irina did research with one of the best time-series econometricians—yes, the one who literally wrote the textbook on it. https://lnkd.in/gE3Xq5dJ

21

1 Comment

A very clean example of building a Data Product in Unity Catalog using a notebook deployed in Databricks Workflow with data quality asserted by DataMesh Manager data contract and CI/CD with Databricks Asset Bundle deployment running in GitHub Actions :) a great way of providing data quality to the Lakehouse 📊📈

47

One of the biggest industry trends is removing ETL pipelines and moving towards ZeroETL world where AWS is leading from the front. Strongly recommend you attend this session where Dipayan and Deep will take us through the details.

36

Happy to share that I've recently completed the extensive Hands-On Essentials: Data Engineering Workshop from Snowflake! 🎓 Key takeaways include: 1. Pipeline Management: Learned the difference between time-driven and event-driven pipelines 2. Production-Ready: Learned how to "productionize" data pipelines, ensuring reliability and scalability 3. Snowflake Tasks and Snowpipe: Gained proficiency in managing Snowflake tasks and Snowpipe for efficient data ingestion and transformation  4. Data Loading Tools and Techniques: Mastered various tools and techniques that data engineers use to load and transform data effectively 5. ETL Pipelines: Leveraged Snowflake to build efficient and scalable ETL pipelines Dived into the essential design principle questions one must answer as a data engineers while building pipelines: 🔹 Is it Self-Running? - Ensuring pipelines execute independently without manual intervention. 🔹 Is it Idempotent? - Guaranteeing pipelines produce the same output on repeated runs. 🔹 Is it Resilient? - Building pipelines that can handle errors and recover gracefully. 🔹 Is it Performant? - Optimizing pipelines for speed and efficiency. 🔹 Is it Cost-Efficient? - Minimizing resource usage for cost-conscious solutions. 🔹 Is it Easy to Maintain? - Prioritizing code clarity and maintainability for long-term success. 🔹 Is it Atomic? - Ensuring data transformations are all-or-nothing, maintaining data integrity. A huge thanks to the Snowflake team for this highly interactive, insightful & fun workshop! 🙌 #Snowflake #DataEngineering #ETL #DataPipelines #DesignPrinciples

22

1 Comment

This read caters to deciphering the #futureofdatascience. Make it a priority to understand these core nuances before diving in as a seasoned #datascientist! Explore the to know more.bit.ly/3ZieX4i

1

Latest blog on how to connect Oracle Autonomous Data Warehouse (ADW) to Apache Superset using the python-oracledb thick client on an Oracle Cloud Infrastructure (OCI) compute instance. Connection between Oracle ADW and Apache Superset can be quite the maze, given the scattered pieces of information out there. This step-by-step guide brings all the necessary details together in one place, making it much easier for you to set up and start exploring your data without much hassle. Thanks to @christopher jones (https://github.com/cjbj). His insights have been helpful for this. https://lnkd.in/g6h5Fi5u #Superset #PythonOracledb #OracleCloud

10