With testing, it's not possible to prove that a system has no bugs. That would take infinite tests, and honestly who has the time? But it only takes one test to *refute* that the system has no bugs. Provided our expectation about what the system's supposed to do in that scenario is *refutable*.
Hillel Wayne
8mo
Writing a 1,000 word essay about this topic at this very second lol
Robin Palotai
8mo
You can actually prove given properties exhaustively for a given range of inputs (the "smallcheck" package in Haskell for example, but admittedly it is quite expensive). Relaxing exhaustiveness, you can do PBT (Property Based Testing), of QuickCheck fame - for Java there's Jqwik which is excellent. Alternatively, you can at least prove properties about a spec, see Lamport's TLA+.
Nándor Tóth
8mo
Karl Popper suggested that scientific theories cannot be conclusively proven but can be refuted by a single contrary finding. Similarly, a software system cannot be proven bug-free by any number of tests, but a single test showing a bug can refute the claim that the system is error-free. So I find your statement quite insightful as it realistically captures the essence of software testing limitations and emphasizes a scientifically sound approach to quality assurance. It reminds us that while we cannot exhaustively test all scenarios, each test serves as a potential refutation of our system's reliability. This highlights not just a challenge but also the importance of crafting clear, refutable expectations in our tests. It underscores the necessity of aligning our testing strategies with well-defined, concrete requirements and objectives to truly gauge and enhance the reliability of our systems.
In short, as I read somewhere "the presence of tests doesn't prove the system has no bugs, but the presence of a bug proves a missing test".
Damian Synadinos
8mo
You might like (parts of) this: https://youtu.be/6_eAv1PNeZY?si=1-QsTHi4HhrNYgbQ
Dr. Michael Köpf
8mo
This holds true way beyond software engineering and very well matches the current philosophy of science that is based on falsifiable hypotheses rather than the futile attempt to prove a theory’s truth.
Hoang Ngoc Anh
7mo
Even with infinite tests, we could never prove a system is bug-free. There are some bugs that cannot be detected just by testing.
Daniel Raniz R.
8mo
I like to think of it like this: every test prooves the non-existence of at least one bug.
Michael Küsters
8mo
While this is mathematically correct, in discussions, people will always argue that you're distracting from the topic by pointing to one specific case where it doesn't work. 🤷♂️
It would not take an infinite number of tests. Here is the proof: You have finite memory. Each cell in memory has finite states (0 or 1). Therefore there are a finite number of states that memory can be in. If you have a single test for each memory state, and that test has a single boolean outcome, you need only a finite number of tests.