Rutuja Redkar

📢 Data Engineering Podcast : Matillion Episode 1 A sincere attempt to bring products/technologies knowledge and awareness to the entire data community out there ! Corneille Marais and SR NAYAK interacted on one of its kind data podcast channel : YourDataShow You would have used Matillion many a times in your data engineering career ! Being an avid user, a solution specialist and being vocal about the engineering products, I absolutely enjoy my conversation on these forums/events : Do you know ? 1> Matillion is a popular modern ELT platform 2> One of the favourites in projects involving ingestion+extraction with Snowflake . Popular snowflake ecosystem tech partner 3> I am so happy to see a local ANZ team out there, had some great conversation which coming in next few episodes. Always great to meet team Ana Yin Corneille Marais Olga Khmelnytska N.B:- Apologies for the audio quality (I underestimated the noise level in background). Promise this will get better and better in upcoming episodes. #Matillion #Snowflake #DataEngineering #YourDataShow

32

6 commentaires

🤯 Gen AI has crazy potential not just for transformation, but for automation and career trajectory. While traditional AI ML come with a demand of coding, data science skill set requirement, Gen AI can turn novice tech enthusiasts into engineers who can tune and deploy models. The reason being, they are pretrained for language - something we’re all comfortable with. 💡In fact, English is touted to be the next big programming language.. ✅ Can you give good instructions? - Great hop on board Sure, proficiency takes time, but considerably easy to get started. And if you mangage to find yourself a neat little role - Try and speak the whole story.. Do not be naive and place all bets on tech ❌If history has taught us anything, it’s that tech has two sides always. For instance, Take the case of LLM or Gen AI While the theoretical use cases remain large, the challenge lies in bias and hallucination. The transformative potential is huge but since these are pretty much non deterministic systems, it’s not cure all and we need to have a human in the loop approach. ⚡There are too many instances of racist bots, biased loan and hiring systems that turned out to be against women, and impact in law and media due to fictional case studies and copy right issues. I sincerely hope all discussions relating to Gen AI transcend the shiny moat, and incorporate what is the need of the hour ✅Developing these tech responsibly and skillng up workforce for the AI era.. Job displacement is an issue but skills like prompt engineering and tool adoption can ensure value added careers rather than mundane backoffice tasks - An area ripe for job displacement. And Indian workforce, whose large part of workday involves data entry, moving info between systems.. I feel are the ones who'll bear the brunt of the axe. PS:- ⚡Gen AI for non sexy office automation - A market of mindblowing proportions waiting for the right players.. Video & image creation is interesting as hell, but it all boils down to the King: ROI #generativeAI #artificailintelligence IBM IBM watsonx #jobdisplacement #automation #responsibleai

25

Studying Continuous Deployment for Data Science (CDDS) on the #KNIME Analytics Platform and #KNIME Business Hub is not just about wanting to obtain an L3 KNIME certification. While the certification serves as validation of the knowledge we've acquired, it encompasses much more. The evolving landscape of the 3 Vs of data—volume, velocity, and variety—demands that we understand how data, #machinelearning, #AI, and #generativeAI are progressing. This understanding is crucial for us to adapt to all these changes. One key point: generative AI will transform our business landscape. Be prepared! #DataScience #KNIME #ContinuousDeployment #MachineLearning #AI #GenerativeAI #DataEvolution #BusinessTransformation #Certification #Adaptability #keeplearning

23

RAG (Retrieval Augmented Generation (RAG)) - fantastic explanation by James Leone in 60 Seconds !! Curious about how Retrieval-Augmented Generation (RAG) is transforming AI? Watch out and learn that in less than 60 seconds !! #AI #RAG #RetrievalAugmentedGeneration #MachineLearning #NLP #AIContentGeneration #AIResearch #AIInnovation https://lnkd.in/dguXw9V7

3

Our clients had a swell time learning about HR Analytics with Victoria O. Odusegun. Her session was so detailed, relatable, practical and insightful! Want a strong start to your HR career by building competence in HR Analytics and other HR functions? Apply today to join HR Primer's next cohort! We have only a few days until we stop taking applications for this cohort, so secure your spot today. See comments for how you can apply. #NewtoHR #HRPrimer #HRCareer #HRAnalytics #HumanResources

5

1 commentaire

Central Data Team vs Business Even after 5 yrs of breakthrough paper by Zhamak Dehghani on #datamesh, the problem between the Central IT Team and Business exists. More apparently, it is evidently clear how deep this problem is! After working on both sides of world, my take on it what led to this situation - 👉 Trust erosion over the period of time between Central Data Team and Business 👉 Resistance to change by the Central data team by the name of standards, time, priority, cost. Hence, easily touted as a “bottleneck”.   👉 Lack of common vision  👉 Sometime, impractical expectations by Business From the world of consulting, the growing trend to have your own Data team, for Business, is visible. The motivation is to have direct control, increased efficiency & business focussed Data team. There is merit in it if the team is relatively small, manageable & co-located to interchange ideas. #Datamesh #datamanagement #dataarchitecture #dataanalytics #bi #data #dataproduct #datastrategy #cdo #dama #datamodeling #dataarchitect #datafabric #dataarchitecture #dataproducts #datavalue #federatedgovernance #dataquality #datagovernance #datamodel

7

2 commentaires

One lesson from Informatica World: otherwise rational humans are willing to have strangers juggle sharp knives inches from their noses 🤔 Here are some other observations. - Informatica’s GenAI copilot for various data mgt tasks will have a lot of experimenters. Users are excited to convert text to SQL code using ChatGPT. They already see the value of similar features in other products, such as GitHub Copilot. - GenAI forces us to redefine data quality. It’s now more than comparing source and target tables or ensuring the consistency of structured datasets. It’s also labeling unstructured data objects, chunking them, and choosing an embedding model to create vectors that accurately represent source documents, imagery, etc. - Moving to the cloud does not save you money on labor costs. Rather, you change your mix of roles: fewer ITOps engineers, more CloudOps, and so on. Data and AI gurus, what you think? Shawn Rogers Peggy O'Neill Mitchell Nitzan Prash Chandramohan #informaticaworld #data #genaiq

50

2 commentaires

Sarath Sasidharan and I have prepared the video for Episode 06, in which we'll study Objective Key Results (OKRs): a goal-setting framework that enhances alignment, engagement, and transparency within businesses, and its pivotal role in managing your data strategy. To ensure a practical understanding, we'll demonstrate using #MicrosoftPurview. We'll guide and define OKRs, enrich them with key results, and link everything to data products. The data itself, is managed within #MicrosoftFabric, so we'll also show that. For subscribing, here's the link: https://lnkd.in/eMw9c-_V

183

6 commentaires

Friday Fun Series ! Well, I come from the era where I have seen hadoop bringing a major shift in data landscape (Of course for good !) and seen its struggle to cope to the growing needs of the data community. In short, the rise and fall left me perplexed ! I feel sad when I saw the concepts like namenode/datanode and the working mechanism did not prosper in long run. I wonder why and here are my thoughts. 1> Non user-friendly interface for usage 2> Complexity around Initial setup and adoption 3> Performance : Need of extensive tuning and monitoring. 4> Cost Efficiency : Managing Hadoop clusters, constant resource allocation and maintenance. 5> Integration : Plug and play was always a challenge and needed extensive setup and management for integrations. 6>Multi-cloud Support : Most scalable data processing/compute products operates on multiple cloud platforms (AWS, Azure, GCP). The growth of hadoop and cloud platform did not go pretty well. 7>Data Sharing , Security and Collaboration : Concentrated , security configuration was always questionable/challenging and do not plug-play easily with all enterprise products easily. 8>Focus on Analytics : Hadoop was initially built for batch processing and handling large datasets, which can make it less suited for real-time analytics Lastly : 9>Evolution/Adaptation : You might have worked in past or have heard of Hortonworks, MapR, Cloudera (still in use with many customers). How many recent customers you came across on day to day basis who are still on MapR/Hortonworks ! The overall design-architecture did not grew with the current needs (on the contrary the cloud eco system growth is phenomenal as its constantly evolving) Snowflake #snowflake #hadoop #bigdata

20

1 commentaire

🌟 Excited to Share My Recent Achievement! 🌟 I’m happy to share that I recently cleared the Snowflake Advanced: Data Engineer Certification! The journey was both challenging and insightful, covering key topics such as performance and cost optimization, working with unstructured data, streams and tasks, Snowpark and much more. What I found particularly valuable were the scenario-based questions that tested practical, real-world skills in areas like loading/unloading data, handling transactions, unstructured data, external functions and shares. More than just a credential, this certification helped me stay up to date with Snowflake’s rapidly evolving feature set. It's a constant reminder to apply best practices in my day-to-day work, ensuring our projects are cost-efficient, scalable, and secure. For fellow Snowflake developers considering this certification, my advice would be: • Focus on optimizing performance and understanding the trade-offs between different data load approaches. Different types of snowflake clients ( python connector , Spark connector, JDBC driver, GO driver etc. ) • Get hands-on experience with unstructured data, Snowpark, data sharing and replication. • Time management - there will be many lengthy scenario based question. Practice some test sets to ensure you quickly skim through the questions, comprehend and select the right option. When in doubt – flag it for review and move to the next question. #Snowflake #DataEngineering #SnowflakeCertifications #ContinuousLearning

31

3 commentaires

Insightful sessions at #informaticaworld #claireAI #mdm #masterdatamanagement #IDMC Informatica Video Credits : Informatica World

17

Creating a data-driven culture is critical to unlocking the full potential of analytics in your business. When everyone from analysts to executives trusts and uses data, the company can innovate and stay competitive. 💡 #DataCulture #Analytics #DigitalTransformation #DataDriven #Data #Data_Analytics #BusinessAnalytics #Business

7

Snowflake Data Governance Framework by Abhijit Debnath 5 Minutes read

10

AIM Research recently collaborated with USEReady to release a research report on the topic - 'Retiring Legacy and Modernizing Analytics Systems Among Enterprises'. This comprehensive survey report explores the transformative shift from legacy BI systems to modern BI systems among large enterprises across various industry sectors. Some of the key findings of the survey report: ⁍ A significant number of organizations (82.8%) recognize the need to migrate from legacy to modern BI systems ⁍ 54.3% respondents are planning to migrate away from legacy to modern BI systems within the next 12 months ⁍ CXOs are deeply involved in migration decisions (94.3%), with 54.3% of the time the decision being solely made by them ⁍ The majority of organizations (68.6%) prefer cloud-based solutions for transitioning their legacy systems to modern BI systems ⁍ Most organizations (70.6%) across sectors expect a small reduction (0-25%) in the size of their analytics teams with the adoption of modern analytics solutions ⁍ Most organizations (77.1%) prefer a phased approach to migration from legacy to modern BI systems Uday Hegde Harjot Singh Kohli Bhasker Gupta Kashyap Raibagi Read the complete report here: https://lnkd.in/eAaiCqXu #SurveyReport #AIMResearch #ModernBISystems #USEReady #LegacytoModernBISystems #AIM

52

1 commentaire

Oh yes 🚀! Happy to have hosted the latest Databricks 'Bytebusters User Community' event 😎! The community took a deep dive into the integration of Databricks with SAP and migrating SAP data to Databricks. Our Senior Data Architect Lothar Verledens captivated the group with his keynote on ‘Next gener-ai-tion migration’. His presentation highlighted the future of data migration, blending AI advancements with seamless data integration. Kudos to Databricks and Fabio dos Reis Brito for the awesome collaboration 🙌. Together, we’re pushing the boundaries of data innovation!  #Databricks #SAP #DataIntegration #DataMigration

29

2 commentaires

In a use case discussion, I came across "create_pbi_agent" in LangChain 🚀 Unlocking the Power of Data Visualization with LangChain and Power BI The fusion of LangChain and Power BI brings a new dimension to how we interact with and derive insights from our data. With LangChain's seamless integration capabilities, querying complex Power BI datasets has never been easier. 🔍 Why is this exciting? Empower your applications to interact directly with Power BI reports and datasets. Enable natural language querying, turning complex SQL-like queries into simple, conversational prompts. Automate insights generation, perfect for decision-making on the go! 💡 Imagine this: Your chatbot or application not only fetches data but also visualizes it in real-time through Power BI. Whether it’s analyzing sales trends, inventory levels, or customer behaviors, the possibilities are limitless. 🎯 Who benefits? From data analysts and developers to business stakeholders, this integration ensures faster insights, simplified workflows, and improved decision-making—all powered by the synergy of AI and data visualization. 📖 Dive deeper and explore how you can build smarter, more intuitive applications using LangChain and Power BI 👉 LangChain Power BI Integration What are your thoughts on integrating AI-driven tools with business intelligence platforms? Let’s discuss! 🤔 Sample Code Snippet - from https://lnkd.in/d7H4RFpR ++++++++++++++++++++++ fast_llm = ChatOpenAI( temperature=0.5, max_tokens=1000, model_name="gpt-3.5-turbo", verbose=True ) smart_llm = ChatOpenAI(temperature=0, max_tokens=100, model_name="gpt-4", verbose=True) toolkit = PowerBIToolkit( powerbi=PowerBIDataset( dataset_id="<dataset_id>", table_names=["table1", "table2"], credential=DefaultAzureCredential(), ), llm=smart_llm, ) agent_executor = create_pbi_agent( llm=fast_llm, toolkit=toolkit, verbose=True, ) ++++++++++++++++++++++ Simple Query on a table agent_executor.run("How many records are in table1?") ++++++++++++++++++++++ #LangChain #PowerBI #AI #DataVisualization #GenerativeAI #BusinessIntelligence

2

Lets think of federated data governance (FDG) like a catchy song. Each data silo is a unique instrument with its own sound. Mature #datagovernance practices/policies/SOPs ensure they play together harmoniously (accurate insights). But with Retrieval-Augmented Generation (#RAG), it's like bringing in a guest musician (external knowledge). #RAG helps the band reference other popular songs (related data) to create an even more catchy, widely appealing tune (deeper insights with wider reach). How do you see #RAG impacting federated data governance? Pour in your thoughts in the comments! #datagovernance #FDG #AI

22

✨ Excited to share my latest achievement! ✨ I recently completed the Snowflake Education Services Hands-On Essentials: Data Engineering Workshop. This workshop provided in-depth insights into modern data engineering practices using Snowflake's powerful cloud data platform. With hands-on experience, I gained practical knowledge in areas like data ingestion, transformation, and optimization, which will greatly enhance my ability to work with data at scale. Looking forward to applying these skills to drive impactful data solutions! 🚀💡 #DataEngineering #Snowflake #CloudData #DataTransformation #ContinuousLearning #ProfessionalGrowth

28

7 commentaires