Eduardo Corazzin

How To Specialize In Data Science / Machine Learning https://ift.tt/RnSuZx4 Is it better to be a generalist or specialist? Photo by Saulo Mohana on Unsplash Eventually, in your data science career, you’ll be asked, “What do you want to specialise in?” It’s quite a daunting question, and knowing what’s best for you is hard to say the least. So, in this article, I will explain why you should specialise, which one is right for you, and how to start. Why should you specialise? In my opinion, you should specialise, but there is no need to rush into this decision. Spend your initial years, like 2–3, learning all the data science and machine learning fundamentals. The things you should cover are: Basic statistics Linear algebra Calculus Linear and logistic regression Random forests Gradient boosted trees Neural networks Support vector machines Modelling best practices: cross-validation, hyperparameter tuning, etc. Writing high-quality Python and SQL This list is non-exhaustive, but it’s a good place to start. It will take a couple of years to fully understand everything. Some may think that’s a long time, but careers are long. If you start as a data scientist in your 20s, you could potentially have four decades working as a data scientist. A couple of years is nothing on that horizon. So, start with the basics and essentially become a generalist. You will then be able to solve most data science and machine learning problems, but you are not necessarily a deep expert in one particular field, which is totally fine at this point. After you have nailed all your fundamentals and have a sound footing, it’s time to decide where you want your expertise to lie in. The specialist vs. generalist debate is years old now, and there are pros and cons to both. Generalist Pros Better for the management track. More job opportunities. Wide variety of work and learning. Stick closer to the business side. Cons Less compensation as not an expert in any one area. Hard to become an IC if that’s something you want to achieve. Harder to get into a research based position. Specialist Pros Deep expertise could be seen as a higher value. Can go down both IC and manager paths. Work in an area you really enjoy. Cons Your expertise may become irrelevant in the future. May have limited opportunities compared to a generalist. I think it’s better to be a specialist because it’s easier to go from a specialist to a generalist than vice versa, and you are likely to be compensated more. However, I don’t think you should be an expert in one niche area. I recommend having T-shaped skills where you know the fundamentals very well and know roughly three areas to a pretty good depth. T-shaped skills diagram by author in Canva. This is kind of the best of both worlds and allows you to be more flexible in the job market. A lot of this assumes you are working in industry. If you want to do research, then it’s probably better to be a deep subject...

1

Awesome list here! If you read Breiman's Two Cultures, be sure to read the version with comments, and you will find an awesome surprise. There is also a full special issue revisiting the paper https://lnkd.in/dVtpsR9f. The following were also some papers that I would add to the list : Roberts, David R., et al. "Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure." Ecography 40.8 (2017): 913-929. https://lnkd.in/dWbAFCmy Rudin, C., Wagstaff, K.L. Machine learning for science and society. Mach Learn 95, 1–9 (2014). https://lnkd.in/dxdXeqAe Cawley, Gavin C., and Nicola LC Talbot. "On over-fitting in model selection and subsequent selection bias in performance evaluation." The Journal of Machine Learning Research 11 (2010): 2079-2107. https://lnkd.in/dtdzmPub Rudin, Cynthia, and David Carlson. "The secrets of machine learning: Ten things you wish you had known earlier to be more effective at data analysis." Operations research & management science in the age of analytics. Informs, 2019. 44-72. https://lnkd.in/dqbtBPyJ Guyon, Isabelle, and André Elisseeff. "An introduction to variable and feature selection." Journal of machine learning research 3.Mar (2003): 1157-1182. https://lnkd.in/dAHxN5dA

16

#data_science_in_Brazil (hoping you are interested in it!😄). Recently, I had the pleasure of connecting with an awesome team of data scientists and exchanging thoughts and knowledge. These are the challenges we discussed: 1. Ethics in Data Science, especially in the era of Facial Recognition capabilities. 2. How to increase trust in AI for the business side and stakeholders—seems like this is a pain point for tech people worldwide. 😅 3. Career Development in Tech. 𝚆𝚑𝚊𝚝 𝙸 𝚛𝚎𝚊𝚕𝚕𝚢 𝚠𝚊𝚗𝚝 𝚝𝚘 𝚍𝚛𝚊𝚠 𝚢𝚘𝚞𝚛 𝚊𝚝𝚝𝚎𝚗𝚝𝚒𝚘𝚗 𝚝𝚘 is that, even though we come from different continents, industries, etc., the questions, challenges, and pain points are pretty much the same. 𝚃𝚑𝚎 𝙿𝚘𝚠𝚎𝚛 𝚘𝚏 𝙳𝚒𝚟𝚎𝚛𝚜𝚒𝚝𝚢 (which isn’t just about gender or geography) is not just a fancy phrase—it is a very powerful tool. When we stay in our little local bubbles, working with the same kind of people, we tend to get stuck in that mindset and miss out on a lot of opportunities. ✨✨ — Jasmin Weimüller and I will be discussing one of these topics in the next episode of our Diaries of a Data Scientist podcast - don’t miss it out!😉 — If you want to know more about the Power of Diversity → I invite you to book your seat by following this link. (http://e-fellows.net/) Patricia Martins de Novais, Dr. Ricardo Klein Sercundes, Luiz Matheus Barbosa, Hismael Costa, Fernanda Aiko Hamatsu, Pedro Ribeiro Monasterio

59

1 comentário

Very interesting keynote talk that I listened to, in person, at the ML in PL conference 2023, about looking at LLMs, not through the lense of sci-fi (Matrix/Terminator), but rather through literature and philosophy https://lnkd.in/dd3CMp6A

6

Hey #datafam Hey everyone! I'm excited to share a project I've been working on: the Data Masking Tool! 🚀 🚀 🚀 Most of my tools originate from real needs within our community. Recently, Ivett Kovács brought to my attention a common challenge we all face: the need to protect sensitive information when working with data. We often require a reliable way to anonymize data for testing and development purposes, as well as creating projects for our Tableau Public based on a real use case we've built for a client, without relying on external services like GPT, which isn't recommended for handling sensitive data. To address this, I developed the Data Masking Tool—a Python application that helps you anonymize sensitive data in CSV and Excel files securely and efficiently. Key Features: 🔒 Secure Data Masking: Replace sensitive information with realistic fake data, all processed locally on your machine. ⚙️ Customizable Options: Adjust prefixes, suffixes, numeric ranges, date ranges, and create custom lists to tailor the anonymization. 🎛️ User-Friendly Interface: An intuitive GUI built with Tkinter makes it easy to select columns and configure settings (The current UI is not great - sorry, I'm not used to creating UIs with Python. Looking forward to collaborating on it 😊). 🔄 Consistent Mappings: Option to keep mappings consistent so that the same original value is always replaced with the same fake value. 📝 Introduce Blanks: Specify a percentage of blank values to simulate real-world incomplete datasets. We often need a reliable way to handle sensitive data without risking exposure. This tool processes everything locally, giving you peace of mind when anonymizing datasets. I'm inviting anyone interested to contribute! 🤝 Open-source projects thrive with community involvement. Whether you want to add new features, fix bugs, or improve the documentation, your help is welcome. 👉 GitHub Repository: https://lnkd.in/dMhvN_mU Let's collaborate to make data handling safer and more efficient. Huge thanks to the Swift Insights Tableau Team for providing feedback. Your feedback and suggestions are greatly appreciated. If you have any questions or ideas, don't hesitate to reach out! Thanks for your support! EDIT: I just added a Jupyter Notebook example in case you prefer it. #dataprivacy #datasecurity #opensource #python #dataanonymization #datamaskingtool #tableau #tableauambassador #hastech

59

🎓 Learnings from the class with Professor Mateus Gerolamo, on Special Topics - Leadership in Data Science at the MBA USP/Esalq: ✨ Leadership and Personality: I discovered that the Trait Theory highlights enduring characteristics that can influence leaders' behavior. It makes me wonder: which personality traits are most relevant for leading teams in Data Science projects? 🤔 ✨ Importance of Leadership: Leadership was emphasized as one of the fundamental factors for achieving results in any human endeavor. 💡 How can I develop my leadership skills to effectively lead Data analysis projects? ✨ Models of Personality Assessment: I learned about the Big Five Personality Model, which is widely used and validated. How can understanding personality traits influence the formation of effective teams in Data Science projects? 🤔 ✨ Organizational Mission: I reflected on the overall mission of an organization, which involves meeting the needs of human beings. How can Data Science projects contribute to meeting the needs of society? 🌍 These learnings inspire me to think more broadly about how leadership, personality, and organizational mission relate to my Data Science and Analytics course at USP. How can I apply these concepts to lead data analysis teams more effectively and in alignment with my organization's values? ✨📊

54

1 comentário

Normalize Your Data with Min-Max Scaling in Python! In the world of data science and machine learning, raw data can have wildly different scales, making it tricky for models to perform optimally. 🔎 What is Min-Max Scaling? It’s a simple yet powerful technique to scale your data into a defined range typically [0, 1]. This ensures all features contribute equally to your analysis or model training. 💡 When to use it? -Data includes features with different units or scales. -Preprocessing for algorithms sensitive to magnitude (e.g., distance-based models like KNN). ✨ Why Min-Max Scaling? Scaled features contribute equally during model training. Simplifies comparison across different attributes. Boosts performance for algorithms that rely on distance metrics. 💡 Pro Tip: Use Min-Max Scaling when you know the range of your data and need all values within a specific boundary. How do you handle varying scales in your datasets? Let’s discuss in the comments! 👇 #Python #DataScience #MachineLearning #DataPreprocessing #Pandas #Normalization #Analytics

16

2 comentários

Hi, Just finished working on an E-Commerce Data which is in SQL format. Did you Know that the city with most delivered product in Brazil is São Paulo? I was able to do Exploratory Data Analysis, to get some insight about the delivered products in Brazil. Some of the approach used are: 1. Import necessary Libraries. 2. Query the Data. 3. Clean the Data. 4. Make Visualization from the data. Here is the link to the notebook. View and Drop your comment. https://lnkd.in/d3f6zyCW

3

2 comentários

I’ve just completed the Introduction to Python Course on DataCamp! Learning a new skill was quick, interactive, and fun! Plus, everything you need is entirely in-browser. Discover which of the 500+ data and AI courses can help you build a stronger career.

2

1 comentário

I had the honor to present my work at the Brazilian Automation Congress (CBA 2024) , where I discussed the combination of system identification with logistic regression for time series classification. This approach aims to help the industry use sensor data to improve safety and maintenance while optimizing decision making. ⚙️📊 It was an enriching experience to share this research with others and receive valuable feedback and insights from renowned experts! Special thanks to Professors Augusto Cerqueira and Erivelton Nepomuceno, whose guidance was essential for the realization and publication of this work. For more information about the event, you can visit the following link, which also includes the pre-proceedings with links to the presented papers: https://lnkd.in/dJEDkxsC #AI #MachineLearning #Innovation #Datascience

39

7 comentários

Enhance your data analysis workflows by integrating AI with PandasAI. This guide reveals how PandasAI simplifies complex data tasks with natural language queries and OpenAI integration. Get step-by-step insights on setting up a secure environment, making your data analysis more efficient and scalable. Automate routine data cleaning and transformation processes to focus on higher-value insights. Visualize data effortlessly with interactive charts and graphs that communicate your findings effectively. Use PandasAI's case studies to see practical applications, from retail to finance, that make analytics more relevant. Unlock new potential in data-driven decision-making by combining traditional data manipulation with AI-powered insights. Ready to transform your analytics? Discover the possibilities with PandasAI today. #EDNAResources #EnterpriseDNA

7

Enhance your data analysis workflows by integrating AI with PandasAI. This guide reveals how PandasAI simplifies complex data tasks with natural language queries and OpenAI integration. Get step-by-step insights on setting up a secure environment, making your data analysis more efficient and scalable. Automate routine data cleaning and transformation processes to focus on higher-value insights. Visualize data effortlessly with interactive charts and graphs that communicate your findings effectively. Use PandasAI's case studies to see practical applications, from retail to finance, that make analytics more relevant. Unlock new potential in data-driven decision-making by combining traditional data manipulation with AI-powered insights. Ready to transform your analytics? Discover the possibilities with PandasAI today. #EDNAResources #EnterpriseDNA

7

Enhance your data analysis workflows by integrating AI with PandasAI. This guide reveals how PandasAI simplifies complex data tasks with natural language queries and OpenAI integration. Get step-by-step insights on setting up a secure environment, making your data analysis more efficient and scalable. Automate routine data cleaning and transformation processes to focus on higher-value insights. Visualize data effortlessly with interactive charts and graphs that communicate your findings effectively. Use PandasAI's case studies to see practical applications, from retail to finance, that make analytics more relevant. Unlock new potential in data-driven decision-making by combining traditional data manipulation with AI-powered insights. Ready to transform your analytics? Discover the possibilities with PandasAI today. #EDNAResources #EnterpriseDNA

1

How To Specialize In Data Science / Machine Learning Is it better to be a generalist or specialist? Photo by Saulo Mohana on Unsplash Eventually, in your data science career, you’ll be asked, “What do you want to specialise in?” It’s quite a daunting question, and knowing what’s best for you is hard to say the least. So, in this article, I will explain why you should specialise, which one is right for you, and how to start. Why should you specialise? In my opinion, you should specialise, but there is no need to rush into this decision. Spend your initial years, like 2–3, learning all the data science and machine learning fundamentals. The things you should cover are: Basic statistics Linear algebra Calculus Linear and logistic regression Random forests Gradient boosted trees Neural networks Support vector machines Modelling best practices: cross-validation, hyperparameter tuning, etc. Writing high-quality Python and SQL This list is non-exhaustive, but it’s a good place to start. It will take a couple of years to fully understand everything. Some may think that’s a long time, but careers are long. If you start as a data scientist in your 20s, you could potentially have four decades working as a data scientist. A couple of years is nothing on that horizon. So, start with the basics and essentially become a generalist. You will then be able to solve most data science and machine learning problems, but you are not necessarily a deep expert in one particular field, which is totally fine at this point. After you have nailed all your fundamentals and have a sound footing, it’s time to decide where you want your expertise to lie in. The specialist vs. generalist debate is years old now, and there are pros and cons to both. Generalist Pros Better for the management track. More job opportunities. Wide variety of work and learning. Stick closer to the business side. Cons Less compensation as not an expert in any one area. Hard to become an IC if that’s something you want to achieve. Harder to get into a research based position. Specialist Pros Deep expertise could be seen as a higher value. Can go down both IC and manager paths. Work in an area you really enjoy. Cons Your expertise may become irrelevant in the future. May have limited opportunities compared to a generalist. I think it’s better to be a specialist because it’s easier to go from a specialist to a generalist than vice versa, and you are likely to be compensated more. However, I don’t think you should be an expert in one niche area. I recommend having T-shaped skills where you know the fundamentals very well and know roughly three areas to a pretty good depth. T-shaped skills diagram by author in Canva. This is kind of the best of both worlds and allows you to be more flexible in the job market. A lot of this assumes you are working in industry. If you want to do research, then it’s probably better to be a deep subject matter expert in a...

2

An interesting talk about intersection between mathematics and computer science (to help on proving theorems). Lean is mentioned as Terence Tao is a heavy user of this programming language developed by a Brazilian guy named Leonardo de Moura. Thanks, Dirceu Barquette Filho, for the recommendation. #mathematics #theoremproving #lean #computerscience https://lnkd.in/daXYfJnG

15

3 comentários

LangChain provides a comprehensive guide to learning Retrieval Augmented Generation (RAG), covering essential aspects like query transformation, HYDE, routing, query construction, indexing, retrieval, generation, and GraphRAG with Neo4j. This resource is invaluable for understanding and applying RAG concepts effectively.

5

The multi AI agent course with crewAI(João (Joe) Moura) from DeepLearning.AI(Andrew Ng) and crewAI will teach you everything you need to know about agentic workflow, from the basics of creating automated task to managing multi agents . By the end of this course, you'll be able to create efficient and effective agentic workflows that will help you improve your productivity and results. I actually implemented it in Streamlit app to understand diagnostic reports of pets- Streamlit (petcare.streamlit.app). Disclaimer - the response is slower as its using open source llm models. #LLM #GenAI #Innovation #ArtificialIntelligence #chatgpt

12

1 comentário

Hi all Data Students! Here are some tips to make your Data Science / Ml portifolio in a advanced/professional way! Avoid starting with Clean / Organized / Beautiful csv files. I in the real world scenario data is messy, unstructured, and sometimes scattered across various sources. So, try to use APIs or web scraping to gather data! Always have in mind the licence for the data ! This will demonstrate your ability to work with dynamic and changing data sources and atention to Data regulation! Perform the EDA! Check outliers and inconsistencies. Deal with missing values, show that you can solve problems with the data! Don't build just a simple model. Take your projects to the next level, apply the best practices such as A/B testing, logging, and hyperparameter tuning. Evaluate different models for the same problem! Tackle Real-World Problems. Focus on projects that address real-world problems or needs. You can find on freelancing platforms like Upwork a real necessity or browse forums to find project ideas. Present Your Findings! I’ve seen some project on GitHub that it is just the code! You need to present the results. If the question / the problem was answered. Create a comprehensive report, or even a video presentation to showcase your work. This will prove that you can bring results to the company! That is may small contribution! If you have more suggestions. Contribute in comments ! Thanks. www.jmcloudpro.com

3