Ashutosh Kumar’s Post

Hi fellow and aspiring #dataengineers !!! End your Friday by learning the concept of "Lazy Evaluation" in #apachespark. I wrote this article last year and till date, it has: 📊 4.95K Views 📊 2.8K Reads with a read ratio of 58%. Enjoy and keep following Ashutosh Kumar for no-nonse content on Data and Software Engineering. Happy Learning !!! https://lnkd.in/gKUgGa79 PS: Medium has great articles and they are detailed too. For learning something, it seems better than LinkedIn. If you are serious about writing/blogging, try on Medium. Account creation is free, I guess. #medium #blogs #loveforwriting #learningandsharing

Day 7 of #100daysoflearning with Ingressive For Good and DataCamp What's the most challenging concept you've encountered so far, and how are you tackling it? The most challenging concept that I have encountered is the concept of Statistics. Probability was a bit easier untill I got to the section of the various distributions, the CDF, Binomial distributions, and RVS, the sane time to use each and how to handle them. I resolved in looking beyond datacamp materials.To Relearn and also to Unlearn. In it all, the most challenging concepts ends up being fun filled.

These recent days, Akela Drissner-Schmid presented a workshop about how to build scalable data ingestion pipelines with Dlt, using open-source tools. This is part of the DataTalksClub LLM zoomcamp program that I currently take. Navigating data ingestion complexities can be hard, but it's crucial for maintaining an up-to-date RAG! So this workshop covered the essential steps: data collection, preprocessing, and timely updates. This was an excellent opportunity to understand how these elements integrate to form a pipeline. I learned how to use dlt to: 1. Extract data from the REST API of Notion pages 2. Load and vectorize into LanceDB, which unlike other vector DBs stores the data and the embeddings 3. Keep the data up to date with incremental loading Here the GitHub link to my experience with these tools and my assignment work: https://lnkd.in/eHqfNiTM In the end, thanks to this workshop, I’ll be able to write a portable, OSS data pipeline for my RAG that I can deploy anywhere, such as python notebooks, virtual machines, or orchestrators like Airflow, Dagster or Mage. #llm #rag #llmzoomcamp2024 #dataengineering #DataTalksClub #DataSkills #OpenAI #Dlt #Lancedb #RestAPI #Embeddings #dataingestion #datapipeline

🚀 Continuing My Unsupervised Learning Series: Mean Shift Clustering! 🚀 I'm excited to share the next part of my Unsupervised Learning series: the Mean Shift Clustering algorithm! 🎓 In this journey, I covered: 🔍 What Mean Shift Clustering is 🏓 How it works using the analogy of marbles moving towards clusters 💡 Kernel Density Estimation (KDE) and its role in the algorithm ⚖️ Advantages and Disadvantages 📸 Applications like image segmentation and object tracking I've also included a simple implementation and compiled everything into a PDF for easy understanding! 📘 Feel free to: 📂 Check out the PDF below 💻 Explore the code on my GitHub repository: https://lnkd.in/gc7aiFJi 💬 Let’s connect and share feedback! ✍️ If anyone is interested in handwritten notes, feel free to ask me! 😊 #MachineLearning #Clustering #MeanShift #UnsupervisedLearning #DataScience #GitHub #LearningJourney #KDE

hey, LinkedIn fam. I've been delving into NumPy Fundamentals, and it's been an enlightening journey! Here are some of the key concepts I've explored: - 📊 Creating NumPy Arrays: Learning the various methods to initialize arrays. - 📏 Array Dimensions: Understanding the importance of shapes and sizes in data manipulation. - 🔢 Data Types: Managing different types of data within arrays efficiently. - 🛠️ NumPy Functions: Leveraging powerful built-in functions for computations. - 🔍 Indexing & Slicing: Accessing and modifying data with precision. - 🔄 Iterating: Efficiently looping through arrays. - 📚 Stacking: Combining arrays to form complex data structures. NumPy has significantly boosted my capabilities in handling and processing large datasets, making my work in data science and machine learning more effective. I am excited to continue this learning journey! 🚀 https://lnkd.in/dVm-wJq8 #DataScience #MachineLearning #Python #NumPy #Programming #TechLearning #BigData ---

🚀 Excited to share that I’ve successfully completed the Supervised Learning course on DataCamp! 🎓 This course provided me with a solid foundation in supervised learning techniques and equipped me with the skills to apply these concepts in real-world scenarios. 💡 A big shoutout to DataCamp for creating such a comprehensive learning experience! 📊 I’m looking forward to leveraging these skills in upcoming projects and continuing my learning journey. 📈 #SupervisedLearning #DataScience #MachineLearning #DataCamp #LearningJourney

Microsoft is thrilled to announce the open-sourcing of GraphRAG, an end-to-end system for dynamically building LLM-powered knowledge graphs using large-scale private data. Watch the announcement video and python codebase below: For a one-click deploy to Azure check out the solution accelerator! #genai #graphrag #microsoft #azureopenai

🌊 Day 278 of #365DaysOfDSA: Solved the Water Flow problem! 🚀 This challenge was all about simulating the flow of water from higher to lower ground, considering multiple sources of water and determining the regions that get wet. It pushed me to think about dynamic scenarios, directional flow, and grid-based traversal. A great exercise in graph theory and flood-fill algorithms! 💧 Every day is a step closer to mastering data structures and algorithms, and I’m loving the journey! Discord Server: https://lnkd.in/gstPhXQX Scaler #DataStructure #Algorithms #CodingChallenge #ProgrammingJourney #CodeNewbieLeetCode LeetCode #AchievementUnlocked #scalerdiscord #codewithscaler #365daysofcodescaler #365daysofcode #Coding #DSA #ProblemSolving #GraphTheory #DailyCoding #Learning #GrowthMindset

A recent project of mine explores the potential of predictive analytics in interpreting survey data. The goal is to predict survey participants' outlooks using their responses, applying a range of data science methodologies. Through this, my aim is to assist in comprehending the intricacies involved in the implementation of such projects. 𝐊𝐞𝐲 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬: • The project is designed within an academic context, emphasizing the exploration of predictive modeling techniques rather than immediate practical application. • Methodologies include comprehensive data preprocessing, train-test split with K-Fold Cross-Validation, grid search for model optimization, and evaluation using metrics like AUC, Accuracy, Precision, Recall, F1, and MCC. • Results and model performances are documented in an accessible format, emphasizing transparency and reproducibility. This endeavor underscores the application of data science techniques in analyzing and predicting human behavior patterns based on survey data. The project's detailed methodology and findings will be available for review and application in the relevant field. Repo link: https://lnkd.in/dHEuA76f #datascience #dataanalytics #python #machinelearning #datamining

⏳ T-minus 1 hour until our Narwhals dev stream! 🚀 Got questions about pandas, Polars, or dataframe interoperability? Bring them to the chat! 💭 https://buff.ly/4geFOWj #OpenSource #DataEngineering