Eduardo "Ed" Contreras, M.S in I.T., M.P.A., B.A

To get started with Poetry as your Python project dependencies and metadata management tool, follow along Giorgos Myrianthous's accessible and thorough guide.

5

Learn how to create pivot tables in SQL for enhanced data analysis — Jack Chang's comprehensive guide covers all the nitty-gritty details you should be aware of.

12

Simplifying Data Workflows with Databricks: Unlock the Power of Apache Spark™️ Databricks is transforming how teams work with Apache Spark™️, offering a unified platform that simplifies data analytics and streamlines workflows. By eliminating infrastructure complexities, Databricks enables data engineers, scientists, and analysts to focus on driving insights, not managing systems. Here’s how Databricks optimizes the Spark™️ experience: 1️⃣ Unified Workspace Databricks creates a collaborative environment where teams can seamlessly work on Spark™️ projects. Supporting multiple programming languages and libraries ensures that teams can use their preferred tools without compatibility concerns. 2️⃣ Automated Infrastructure Databricks automates Spark™️ cluster provisioning and scaling, removing the need for manual configuration. This allows teams to focus on analysis and innovation while ensuring optimal performance. 3️⃣ Performance Optimization With features like intelligent caching, query optimization, and automatic tuning, Databricks boosts Spark™️ workload efficiency, leading to faster query execution and reduced resource costs. 4️⃣ Integrated Ecosystem Databricks integrates with tools such as Delta Lake, MLflow, and TensorFlow, enabling comprehensive data pipelines and enhancing advanced analytical capabilities. This integration accelerates the time to insight and simplifies data management. By streamlining the Apache Spark™️ experience, Databricks empowers organizations to unlock the full potential of their data and accelerate business outcomes with greater efficiency. #Databricks #ApacheSpark #BigData #DataScience #MachineLearning #UnifiedAnalytics #DataEngineering #CloudComputing #AI #BusinessIntelligence #outsourcing #outsourcingservices #grapplesoft

3

Databricks Apps is in public preview, and it's awesome. Here are 5 things we love about it...and why 👇 𝗨𝗻𝗽𝗮𝗿𝗮𝗹𝗹𝗲𝗹𝗲𝗱 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗦𝗶𝗺𝗽𝗹𝗶𝗰𝗶𝘁𝘆 • Leverage familiar Python frameworks: Build with Dash, Shiny, Gradio, Streamlit, and Flask. • Seamless IDE integration: Develop in Visual Studio Code or PyCharm, your choice. • Pre-built templates: Jumpstart your app development with ready-to-use Python templates. • Local or cloud development: Work in your preferred environment, then deploy effortlessly. 𝗖𝘂𝘁𝘁𝗶𝗻𝗴-𝗘𝗱𝗴𝗲 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 • Serverless compute: Apps run on auto-provisioned infrastructure, eliminating manual setup. • Git integration: Benefit from version control for collaborative, trackable development. • CI/CD pipeline support: Streamline your workflow with integrated continuous integration and deployment. • Scalability: Automatically handle varying workloads without manual intervention. 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲-𝗚𝗿𝗮𝗱𝗲 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗮𝗻𝗱 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 • Unity Catalog integration: Centralized data governance across your entire Databricks ecosystem. • Granular access controls: Ensure data is accessed only by authorized personnel. • Automated service principals: Simplify and secure app-to-service authentication. • SSO and OIDC/OAuth 2.0: Robust user authentication mechanisms built-in. 𝗦𝗲𝗮𝗺𝗹𝗲𝘀𝘀 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 • Access to Databricks SQL: Harness the power of SQL queries directly within your apps. • AI model serving: Integrate machine learning models seamlessly into your applications. • ETL job integration: Connect your apps to data pipelines for real-time insights. • Unified data access: Leverage Delta Lake, data warehouses, and data lakes effortlessly. 𝗘𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺 𝗘𝘅𝗽𝗮𝗻𝘀𝗶𝗼𝗻 𝗮𝗻𝗱 𝗩𝗮𝗹𝘂𝗲 𝗖𝗿𝗲𝗮𝘁𝗶𝗼𝗻 • Databricks Marketplace: Partners can now build, package, and sell data products and AI assets. • Self-service analytics: Empower business users with custom, governed data applications. • Rapid prototyping: Quickly iterate on ideas and bring concepts to life. • Cross-functional collaboration: Bridge the gap between data teams and business stakeholders. Yes. We partnered up with Databricks. But we didn't 𝗵𝗮𝘃𝗲 to. Innovations like Databricks Apps make us just 𝘄𝗮𝗻𝘁 to. 🙂

14

✅ Data Literacy ✅ Data Fluency ✅ Data Confidence Whatever the terminology your organization prefers, enabling and educating data essentials is key to pushing your program forward. What other areas would you include in your data enablement program? 👌Come learn and earn with the community ~ DataQG.com

13

1 Comment

I like to think of MERGE statements as performance problems waiting to happen. They remind me of some computer games I used to play when I was younger, like Baldur's Gate (the original!) or Mech Commander. You could control your characters in a group, or as individuals. If you controlled them as a group, the computer used its "intelligence" to do the routing for you as the group moved. Lord help you if you encountered that complex object known as "a door" because all your people would bunch up in it and shift positions, taking 30 seconds to 5 minutes (and sometimes never) to get through. But if you selected each character individually and routed them through the door, it worked flawlessly. Simplify your ask of the database engine as much as possible, and you'll get better, more consistent results. It takes a few more lines of code for sure, but it will turn out better. There are some great links and opinions in the post below about the reasons to avoid MERGE.

1

4 Comments

💡 5 Tips for Conducting Effective Technical Interviews for Data Engineers 💡 Are you struggling to find the right data engineering talent? Look no further! At Dataforce, we specialize in connecting you with top-tier data engineers for both contract and permanent positions. Here are our top 5 tips to ensure your technical interviews are on point: Define Clear Requirements 📝 Clearly outline the technical skills and experience required for the role. This helps in setting the right expectations and focusing the interview on relevant areas. Use Real-World Problems 🌍 Design interview questions around real-world problems that your company faces. This not only tests the candidate's technical skills but also their problem-solving abilities and how they approach practical challenges. Evaluate Communication Skills 🗣️ Ensure candidates can explain their thought process and solutions clearly. Data engineers need to collaborate with various teams, so strong communication skills are crucial. Incorporate Hands-On Tasks 💻 Use coding challenges and live coding sessions to assess candidates' hands-on abilities. Platforms like HackerRank and LeetCode can be useful for this purpose. Assess Cultural Fit 🤝 Cultural fit is as important as technical skills. Ask questions that help gauge the candidate’s alignment with your company's values and work environment. 🔍 Need help finding the perfect fit? At Dataforce, we take the guesswork out of hiring by providing you with pre-vetted, highly skilled data engineers ready to tackle your toughest challenges. Whether you need contract or permanent hires, we’ve got you covered! Let’s connect and discuss how we can support your hiring needs. 🚀 #HiringTips #DataEngineering #TechnicalInterview #Recruitment #TechHiring #DataScience #Dataforce

3

Company executives continue to demand their data teams turn their data into gold...all while pressuring data teams to push out quick and dirty pipelines. There isn't time to actually understand the data they are modeling. There isn't time to actually understand if the data they are using is even accurate. Nope, just deliver, deliver, deliver. As companies look to incorporate AI and ML into their data strategies and roadmaps the problem could compound. BUT...I look at this as a new opportunity to refocus on data quality. Regardless of how fancy or sophisticated a company's AI model might be, poor data quality will make the outputs of these models useless at best, and misleading and company destroying at worst. So, as your company is rolling out its internally developed LLM or implementing a dynamic pricing model, it’s a great time to review your data quality strategy. The first key point to cover is that data quality is not limited to one pillar, like accuracy. But instead, several key pillars need to be considered when developing your data quality system. So in the article below, I cover some of the key pillars that your company should consider as you're building your AI and ML infrastructure. https://lnkd.in/gscnmc5D

46

Stop Abusing SELECT * – Your Database Is Crying! 😅 I get it. It’s easy. It’s fast. But every time you write SELECT * FROM, a DBA somewhere sheds a tear. 💧 Think of your database as a fine dining restaurant 🍽️ – you wouldn’t order everything off the menu, right? (Well, maybe if you're really hungry, but stay with me! 😂) Instead, you pick exactly what you need. SQL queries work the same way! 𝐖𝐡𝐲 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐞? - Your queries will run faster than coffee through your system on Monday morning. - You’ll save your company $$$ by not wasting resources like RAM, CPU, and your DBA's sanity. - Your database will thank you. (Okay, maybe not out loud, but you’ll feel the love. 😄) Dos & Don'ts of SQL – AKA How to Keep Your Database Happy: ✅ 𝐃𝐎: Ask for what you need, like a polite person. SELECT name, age FROM – your DB will thank you. Index like it’s the secret ingredient to grandma’s lasagna – a little goes a long way. Use joins. Because no one wants to be the lone table at the data party! 🚫 𝐃𝐎𝐍’𝐓: SELECT * FROM – Unless you’re feeling lucky and want to stress-test your servers (not recommended). Ignore WHERE clauses. Let’s be honest, filtering is key – even for that mountain of emails. 𝐔𝐬𝐞 𝟏,𝟎𝟎𝟎 𝐬𝐮𝐛𝐪𝐮𝐞𝐫𝐢𝐞𝐬 𝐭𝐨 𝐠𝐞𝐭 𝐚 𝐬𝐢𝐦𝐩𝐥𝐞 𝐚𝐧𝐬𝐰𝐞𝐫. 𝐀𝐢𝐧’𝐭 𝐧𝐨𝐛𝐨𝐝𝐲 𝐠𝐨𝐭 𝐭𝐢𝐦𝐞 𝐟𝐨𝐫 𝐭𝐡𝐚𝐭! --- So, the next time you're tempted to SELECT *, remember, you’re not just querying data—you’re also keeping your database from an existential crisis! 😅 𝐀𝐧𝐲𝐨𝐧𝐞 𝐞𝐥𝐬𝐞 𝐠𝐮𝐢𝐥𝐭𝐲 𝐨𝐟 𝐒𝐄𝐋𝐄𝐂𝐓 * 𝐅𝐑𝐎𝐌 𝐜𝐡𝐚𝐨𝐬? 𝐃𝐫𝐨𝐩 𝐲𝐨𝐮𝐫 𝐜𝐨𝐧𝐟𝐞𝐬𝐬𝐢𝐨𝐧𝐬 𝐛𝐞𝐥𝐨𝐰. 👇 𝐋𝐞𝐭’𝐬 𝐭𝐚𝐥𝐤 𝐨𝐩𝐭𝐢𝐦𝐢𝐳𝐚𝐭𝐢𝐨𝐧!!

14

1 Comment

Is your company’s big data strategy with centralized systems leading to data quality and accessibility issues? EXL’s VP of Data, Analytics & AI, Dave Crolene, delves into how moving to a distributed data architecture will facilitate flexibility and real time insights. In his recent article for CIO Online, Crolene describes how decentralized data allows business units to innovate independently while maintaining unified governance. Discover the importance of transitioning to a distributed data architecture especially as generative AI disrupts technology. https://bit.ly/4eBj9Sr #Data #AI #DataArchitecture #GenerativeAI

118

Break free of the traditional approach to data migration and learn how organizations are accelerating their modernization timelines. Tap into better outcomes, higher ROI, and the power of the Snowflake AI Data Cloud. https://lnkd.in/gUUPz8fC #datamigration #snowflakemigration #snowflakeconsulting

27

SQL Question 46: Subscription Overlap Tracker Full Video: https://lnkd.in/guQA4iEQ 0:00 - Introduction to dataset and Question 4:00 - Approach 1: Using Date Range as Filter 08:40 - Approach 2: Simplified Date Filter 10:15 - Approach 3: Correlated Subquery 12:40 - Conclusion Video 249: This is the 46th video of the SQL Interview Question series. We are given a table of product subscriptions with a subscription start date and end date for each user, write a query that returns true or false whether or not each user has a subscription date range that overlaps with any other completed subscription. Completed subscriptions have end_date recorded. In this video, I explain how to identify overlapping subscription periods for each user from a subscription table. *** Approach 1: Using Date Range as Filter ***   In this method, we use a self-join on the subscription table and apply a date range filter to detect overlaps. We leverage a LEFT JOIN to compare each subscription's start and end dates with all others, excluding the same user. By using conditional aggregation, we determine the presence of overlaps. *** Approach 2: Simplified Date Filter ***   This approach simplifies the date filter conditions to check if the start date of one subscription is before the end date of another and vice versa. Similar to the first approach, we use a LEFT JOIN and conditional aggregation to identify overlaps. *** Approach 3: Correlated Subquery ***   In this method, we use a correlated subquery within a CASE statement to check for overlapping subscriptions. This approach avoids the use of joins and instead checks the overlap condition for each row using a subquery. For a comprehensive understanding of these SQL methodologies and their application, please refer to this explanatory video. code: https://lnkd.in/g3GAmCgi Follow me on, Website : https://growwithdata.co/ YouTube : https://lnkd.in/gSrqE3Nj TikTok : https://lnkd.in/gqFiMex2 LinkedIn : https://lnkd.in/gdfNNX9a Facebook : https://lnkd.in/gU6JWpFH FB Group : https://lnkd.in/gG4ENcrP twitter : https://lnkd.in/geZ3dqi8 Instagram : https://lnkd.in/gn263_PZ WhatsApp : https://lnkd.in/gBa5eC7y TheWide : https://lnkd.in/gt7htz_8 #sql #dataengineers #tablejoins #ceil #floor #bucket #meta #google #facebook #apple #paypal #netflix #amazon #deinterview #sqlinterview #interviewquestions #leetcode #faang #maanga #mysql #oracle #dbms #query #sqlserver #mysql #coderpad #aggregates #aggregation #nonaggregation #database #placementpreparation #lead #lag #windowsfunction #nullcheck #coalesce #sqlperformance #ifnull #case #lead #lag #windowsfunction #tamil #tamilpython #tamilinterview #tamilinterviewlatest #tamilinterviewquestions #sqlintamil 

1

Who uses Sync? Organizations that invest in optimizing Databricks spend❗ Gradient by Sync uses an advanced machine-learning algorithm to ensure that your team's Databricks jobs are meeting and exceeding business objectives, like runtime and SLAs. 🤯 The best part, you don't have to change a single line of code. If you're spending tens of hours per week per developer to optimize your cloud compute infrastructure, we can help save those engineering hours and cut costs. Learn more about Gradient by Sync here: https://hubs.ly/Q02Khl4d0

3

One thing that you occasionally want to do in (Spark) SQL is select all columns with the exception of a few. The traditional SQL way is to replace a SELECT * with an exhaustive list of columns. I was looking for a handy dandy way to expand the * so I could snip out the few columns I didn't want. Imagine my surprise when I found this construction: SELECT * EXCEPT(columns, I, want, to, exclude) FROM sometable It beats manually specifying each column or jumping over to pyspark just to drop a few columns.

30

3 Comments

At AgileData we love sharing patterns and patterns templates that help data teams improve their Way of Working to add more WoW to their day. This is a great overview of the different data maturity assessments patterns that are out there, have at it! Don’t forget to reach out to Nicole Janeway Bills and team and let them know if you end up using it and got value.

3

3 Comments

A Data Management Maturity Assessment (DMMA) is a structured evaluation designed to help you assess your organization's data practices across multiple areas. My team and I put this graphic together to help you compare them. #mdm #assessment #datagovernance #datamanagement #data

56

1 Comment

🧠 Retention Periods: Not Just for Backups! 💾 Have you ever returned to work after a few days off and felt like you forgot everything about your job role—like your SQL DBA skills just vanished? 🤔 The truth is, {{our brains have retention periods}} too, much like the retention periods we set for backups and data files. When we don’t actively use certain skills or knowledge, our brain starts to deprioritize them, leading to that "forgotten" feeling. But don’t worry—just as we can restore data from backups, a quick review can refresh our memory and bring those skills right back! 🔄 #SQLServer #DBA #TechTips #MemoryRetention #Backups #Productivity #Data #Engineer

5

1 Comment