DoltHub’s Post

Get Your Data Ducks in a Row Mastering Test-Driven Development (TDD) with code that touches a data source feels like a tightrope walk, but here’s the gear you need: start by designing your code with testability in mind—separate concerns and leverage Dependency Injection. Mock and stub your data sources to control and simulate various scenarios effortlessly. Opt for in-memory databases when realism and speed are key. Set up structured fixtures and data factories for consistency, and don’t forget to maintain a suite of integration tests in a dedicated environment. Embrace CI/CD automation to catch issues early and ensure consistent testing environments. Keep your test data version-controlled and maintain a known state for each test. Follow these steps, and you’ll navigate the TDD tightrope with ease. -//- 1. Design with Testability in Mind Separate your business logic from data access logic. Use design patterns like Repository or Data Mapper and Dependency Injection to make substituting real data sources with test doubles easier. 2. Use Mocking and Stubbing for Data Sources Utilize mocking frameworks to create mock objects or stubs of your data sources. This allows you to simulate various scenarios and control data source behavior during tests. 3. Create In-Memory Databases for More Realistic Tests Use in-memory databases for tests that need more realistic interactions with a database. Populate them with predefined data tailored for your tests for consistency. 4. Use Fixtures and Factories Set up test fixtures to prepare the necessary environment and data before each test. Implement data factories to generate required data objects, avoiding hard-coding test data. 5. Implement Integration Tests Separately While unit tests should be isolated, maintain a suite of integration tests to verify interactions with actual data sources. Use a separate environment for these tests. 6. Use CI/CD for Automated Testing Integrate your tests into a CI/CD pipeline to run automatically on each commit, catching issues early. Use containers to ensure consistent testing environments. 7. Maintain Clear Test Data Management Keep your test data in version control to ensure it evolves with your application. Ensure tests don't depend on the external database state, starting each with a known, consistent state. #TDD #TestDrivenDevelopment #UnitTesting #Mocking #SoftwareDevelopment #DevOps #ContinuousIntegration #InMemoryDatabases #BestPractices #CodeQuality

🔧 Tackling legacy codebases can be a daunting task, especially when testing frameworks are absent. In our new blog post, we outline effective strategies for initiating testing in these challenging environments. Here’s what you can expect to learn: 1️⃣ **Identify Business-Critical Areas**: Focus on key modules for maximum impact. 2️⃣ **Utilize Static & Dynamic Analysis**: Enhance your testing by leveraging analysis tools. 3️⃣ **Develop a Comprehensive Testing Strategy**: Prioritize modules and automate builds to streamline processes. Join us as we provide practical insights from experienced developers to help improve your code quality! #LegacyCode #SoftwareDevelopment #TestingStrategies #QualityAssurance #TechInsights

It’s good to see the proposal and extending integration of o11y to CI/CD automation #devops

Hello, LinkedIn Connections 🤩 ! Dive into my latest blog Published in HashNode to explore the impact of Behavior Driven Development (BDD) on creating highly robust and fault-tolerant software. Discover the advantages and limitations of BDD and join the discussion on whether it's a 𝐛𝐨𝐨𝐧 𝐨𝐫 𝐛𝐚𝐧𝐞 𝐟𝐨𝐫 𝐬𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭. Let's explore together! #GWM #GrowWithMe #BehaviourDrivenDevelopment #SoftwareDevelopment #Agile #DevOps #AutomatedTesting #CICDpipeline

𝐃𝐨𝐜𝐤𝐞𝐫 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 _______________________________________ 𝐃𝐞𝐯𝐎𝐩𝐬 𝐅𝐑𝐄𝐄 𝐖𝐞𝐛𝐢𝐧𝐚𝐫 & 𝐑𝐞𝐬𝐨𝐮𝐫𝐜𝐞𝐬 https://lnkd.in/e7hnkvSa _________________________________________ 𝐈𝐧𝐭𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧 𝐭𝐨 𝐃𝐨𝐜𝐤𝐞𝐫: ● Docker is essential for modern application deployment. ● It's popular among developers and system administrators for its efficiency. 𝐃𝐨𝐜𝐤𝐞𝐫 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞: ● It follows a client server model. ● Components include Docker Client, Docker Daemon, Docker Images, Docker Containers, and Docker Registry. 𝐃𝐨𝐜𝐤𝐞𝐫 𝐂𝐥𝐢𝐞𝐧𝐭: ● Interface for users to interact with Docker. ● Sends commands to the Docker daemon for execution. 𝐃𝐨𝐜𝐤𝐞𝐫 𝐃𝐚𝐞𝐦𝐨𝐧: ● Runs on the host machine. ● Executes tasks requested by the Docker client, like building and running containers. 𝐃𝐨𝐜𝐤𝐞𝐫 𝐈𝐦𝐚𝐠𝐞𝐬: ● Templates for containers. ● Contains all dependencies needed for applications. 𝐃𝐨𝐜𝐤𝐞𝐫 𝐂𝐨𝐧𝐭𝐚𝐢𝐧𝐞𝐫𝐬: ● Running instances of Docker images. ● Contains code, runtime, libraries, environment variables, and config files. 𝐃𝐨𝐜𝐤𝐞𝐫 𝐑𝐞𝐠𝐢𝐬𝐭𝐫𝐲: ● Repository for Docker images. ● Docker Hub is a public registry, but private registries can be created. 𝐁𝐚𝐬𝐢𝐜 𝐃𝐨𝐜𝐤𝐞𝐫 𝐂𝐨𝐦𝐦𝐚𝐧𝐝𝐬: ● 𝐃𝐨𝐜𝐤𝐞𝐫 𝐏𝐮𝐥𝐥: Checks if the image is locally available. Downloads from Docker Hub if not. ● 𝐃𝐨𝐜𝐤𝐞𝐫 𝐁𝐮𝐢𝐥𝐝: Builds Dockerfile to create local images. Dockerfile contains build instructions. ● 𝐃𝐨𝐜𝐤𝐞𝐫 𝐏𝐮𝐬𝐡: Uploads image to Docker Hub. Share images with others. ● 𝐃𝐨𝐜𝐤𝐞𝐫 𝐑𝐮𝐧: Run a container from an image. Executes image's executable and exposes ports. Useful for starting various applications. Credit - Omkar Srivastava

When everything is "As Code" today Why not "test data as code." Challenge Overview: The foremost challenges in contemporary software development involve creating, administering, disseminating, and keeping test data. The reliance on data leads to immutable environments, escalating cloud costs. Companies grapple with test data impediments, hindering testing activities without access to essential historical data. The realization of a fully automated CI/CD pipeline is obstructed without the automation of data requirements. Solution: Establishing a centralized team with representation from diverse departments is paramount to address this challenge. This team assumes responsibility for defining processes, technical architectures, solution building, reporting, and governance. A technical solution involves developing a seed data loader triggered automatically from the code. During sprint planning meetings, data requirements for each user story are identified and converted into Data as Code, which is then pushed to a Git repository. Implementing a Data as a Service microservice facilitates fetching files from a centralized Git repository and populating data on any environment as needed. This service, accessible as a pipeline job, caters to various data requirements across the company. Encouraging a culture of building and managing data as code is vital in the current landscape, where everything is maintained as code. The benefits of adopting Test Data as Code include: • Promoting immutability • Expediting CI/CD processes • Ensuring data governance and security • Enhancing Scrum Team Productivity Critical Goals for Efficient Test Data Management in CI/CD: 1. Develop an approach to treat the dataset as a service or code, expediting data loading on any environment. 2. Enable creating and teardown environments without dependency on specific datasets. 3. Foster consistency across teams in the processes employed for managing data. By embracing Test Data as Code and aligning with these goals, companies can streamline their CI/CD pipelines, promoting agility, consistency, and efficiency in managing test data. #testing #testdata #devops #engineering #cracktechnicalinterview

In the fast-paced field of #softwaredevelopment, ensuring applications remain functional and secure through updates is essential. The new post from #Kondukto explores the importance of #DAST tools, integration, and enhancement in #regressiontesting. https://lnkd.in/e_7CV8_j

🎉🎉🎉 RELEASE ANNOUNCEMENT 🎉🎉🎉 Testing with Celery has *never* been so easy! I am glad to announce the release of a completely new plugin for pytest, pytest-celery, that provides E2E testing capabilities for Celery Applications like never before 🤯 This new release has been under my umbrella for about a year, and I am excited to finally share the fruits of my labor with the world! User Manual: https://lnkd.in/eFDUA-kg PyPI: https://lnkd.in/efzENYBy Source: https://lnkd.in/ev6ZHJrQ Can be used for Unit, Integration and Smoke / Production-like testing environments. Install with: pip install -U "pytest-celery[all]" ** Spread the news! ** Key Highlights - Simple: TDD-based design for simple API usage. - Flexible: Highly configurable for even particular cases. - Fast: Fully supports pytest-xdist. - Annotated: IDE autocomplete support for the plugin’s API. Supported Vendors - Brokers: RabbitMQ, Redis - Backends: Redis, Memcached Main Features - Architecture Injection: Allows dynamically injecting architecture components. - Docker-based: Uses Docker behind the scene. - Batteries Included: The plugin provides a set of built-in components that can be used to test ideas quickly. - Code Generation: Mechanism for advanced testing capabilities at runtime. - Isolated Environments: Tests don’t mess up each other, yes, even if you forgot to clean up. - Tests as First-Class Citizens: The plugin is designed to work for you, not the other way around. - Extensible: Following the S.O.L.I.D principles, the plugin uses a plug-and-play design to allow easy replacement of pipeline components.

What is API Mocking? API Mocking involves creating an environment that mimics the behavior of a real and functional API. This allows the clients to test different scenarios and do not have to wait until the completion of API development. There are number of benefits using API mocking. 1. Faster Feedback 2. Cost Savings 3. Increased Security 4. Improved Reliability 5. Cross-Platform Compatibility There are various number of API Mocking tools and below are few examples I found in the blog https://lnkd.in/gvrp4KpV 1. Mockoon 2. Stoplight 3. Postman 4. Mountebank 5. Apiary 6. MockServer 7. Hoverfly 8. WireMock 9. Prism 10. Mocky

CI/CD in #dbt CI (Continuous Integration): Test your changed models when a PR is created. - You create a Feature Branch changing model_b - You create a PR to main with those changes - The PR triggers a GitHub action (or bitbucket pipelines, etc) - The action creates a dedicated CI schema to run <dbt build -s 'state:modified+' --defer --state path/to/artifacts --target ci_schema> - The command will run and test the modified (and downstream) models in the dedicated schema - The PR can be approved if all of them are built successfully. - The dedicated schema can be dropped. CD (Continuous Deployment): Build your modified models when a PR is merged. - You merge the previous PR into the Main Branch. - The PR triggers a GitHub action (or bitbucket pipelines, etc). - The action creates a dedicated CI schema and runs <dbt build -s 'state:modified+' --state path/to/artifacts --target prod> - This invocation updates the modified (and downstream models) models in production. About the invocation flags: --state: Defined the path where the prod artifacts are located --target: Defines the connection to be used. In this case, we change targets to change the schema where models will be written. --defer: I explain defer in this post: https://lnkd.in/dcE9djDd This repo has an implementation of CI/CD with dbt-core and BigQuery https://lnkd.in/dxY8_AM6 Follow me for daily dbt content 🔶 #analyticsengineering #dataengineering