Skip to content
Close
SEARCH
Login
Login

Best Practices for Java Testing

Testing Java applications for security vulnerabilities is crucial to prevent downtimes.

Secure Your Java Apps

Code Intelligence leverages the best of static and dynamic application security technologies, including advanced fuzz testing, to achieve maximum code coverage without false positives. Try it out in our fuzzing tool CI Fuzz.

Try CI Fuzz
ci-sense-new-feature-dashboard_files

What to Expect on This Page

Why Java Vulnerabilities Are So Dangerous

A large part of our lives happens on the internet. As it happens, a lot of it is written in Java (how much exactly is frequently discussed). Traditionally Java applications were large and monolithic. Within modern Java applications, there is a clear trend towards microservice architectures that are connected via web APIs (mostly REST APIs). Unlike embedded applications, the challenge in java testing is not to secure the memory access, but the immense complexity that arises from the connectivity between the modules and APIs that connect them.

If vulnerabilities in dependencies of Java applications are exploited, they can be used to maliciously spread data throughout multiple microservices rapidly. Since many microservices fully trust inputs from internal sources, attackers can often access large parts of the entire web backend through a small entry point. This was most recently shown by the RCE found in the popular Java library log4j, which affected thousands of Java applications around the world.

While modern microservice architectures are easier to maintain and more fail-safe, their complexity leaves little room for error. Accordingly, applications need to be tested thoroughly and with java testing tools that can deal with their complexity.

Is Java More Secure Than C/C++?

Memory-safe languages such as Java, Golang, or Python get their names from their runtime error prevention mechanisms. Compared to C/C++, this basically makes memory-safe languages immune to memory corruption. Combined with the fact that memory-safety vulnerabilities make up a large portion of all vulnerabilities (e.g., 90% of Android vulnerabilities), this can lead to the impression that Java is much more secure than C/C++. However, this statement has to be treated carefully. Although the density of vulnerabilities in Java is far lower than in C/C++, Java vulnerabilities can have devastating consequences such as downtime or data theft.

Common Java Vulnerabilities

While C/C++ vulnerabilities are frequent but generally easier to find, Java vulnerabilities are often more intricate. The OWASP foundation maintains a list of the ten most dangerous web vulnerabilities. Java applications are susceptible to most of them. Here are three particularly harmful bugs that are often found in Java applications.

SQL Injections

During an SQL injection (SQLi) or a similar injection attack, attackers insert malicious SQL statements into a system through an input field (e.g., username or password). If the application fails to sanitize such inputs, the statement will be executed, which enables attackers to access the database. 

The term SQL injection can be divided into several sub-categories, including in-band, out-of-band, error-based,and union-based SQLis. One of the best ways to prevent SQL injections is by validating user inputs thoroughly, using filters, and testing for vulnerabilities that could expose the database to arbitrary commands.

Remote Code Execution

A remote code execution (RCE) is a software attack during which attackers exploit software vulnerabilities to gain control of someone’s computing device, usually by tricking victims into downloading malware. This allows attackers to steal data, monitor devices, divert funds, and other malicious activities. To prevent RCEs, it is considered best practice to implement firewalls, alert systems, and security testing measures that can detect potentially harmful vulnerabilities.

Cross-Site-Scripting

Cross-site scripting (XSS) is a widespread security vulnerability in Java applications. When exploited, It allows attackers to inject malicious code into web pages. The code is then executed in the client's web browser. By using XSS, attackers do not target a victim directly but rather a vulnerability in a website or web application that the victim visits or uses. Thus, the vulnerable website essentially serves as a channel for the attacker to send a malicious script to the victim's browser. Cross-site scripting can be prevented by escaping and validating user inputs and by testing web applications for vulnerabilities that could be exploited.

The Most Spectacular Java Bug of the Century: Log4j

An example that demonstrates the importance securing Java applications is the RCE (CVE-2021-44228) that was found in the open-source library log4j in December 2021. Since log4j is a highly popular logging utility, the disclosure of the RCE affected countless applications and over 3 Billion end devices. In early January, yet another RCE vulnerability was found in log4j, although this one was not as severe as CVE-2021-44228.

After the log4shell (CVE-2021-44228) vulnerability was patched with version 2.15, another CVE was filed. Apparently, log4j was still vulnerable in some cases to a denial of service. However, it turned out that on some systems, the issue can still lead to a remote code execution. In this video, LiveOverflow uses the Java fuzzer Jazzer to find a bypass.

Try Out CI Fuzz

CI Fuzz is a fuzzing tool that makes fuzz testing as easy as unit testing. It allows you to automatically test your Java applications without leaving your preferred dev environment. CI Fuzz is compatible with Bazel and Maven and comes with an integration into JUnit. If you've ever run a unit test, you will be able to use it.

Try CI Fuzz

Common Approaches to Java Testing

Similar to other programming languages, there are multiple approaches how to test a Java application, each one of them possessing its own strengths and weaknesses. The following paragraphs aim to elaborate on the characteristics of some of the most common Java testing approaches. It is important to mention that many of these testing methods complement each other and can be implemented together.

Unit Testing for Java

Unit testing is done to test an individual software unit by ideally testing every method that it offers. Most unit tests use a predefined set of inputs, to execute the functions of the system under test. The outputs and the behavior of the application are then observed to determine whether a unit test fails or passes. In many development teams, test cases for unit tests are still created manually, which is highly time-consuming and not particularly accurate. Apart from functional and security testing, unit tests are also frequently used to drive software design.

Static Application Security Testing for Java (SAST)

During static application security testing, the software under test is scanned without executing it to analyze its underlying framework. Since DAST is a white-box testing approach, it requires access to the source code. Opponents of SAST criticize that it produces many false-positive test results that have to be sorted out manually. SAST tools are most efficient when implemented alongside more specialized java test tools that can analyze the structure of Java applications and discover runtime errors. 

Dynamic Application Security Testing for Java (DAST)

Dynamic Application Security Testing is a testing method during which randomized or predefined test inputs are fed into a system, with the goal of triggering faulty program states. DAST is often conducted as a black-box testing method and therefore does not require access to the source code. Since DAST tools test the software under test from the outside-in, it is a great method to test java code from the attacker's perspective.

Fuzz Testing for Java

Fuzz testing is an increasingly popular solution for testing Java code. As opposed to a predefined set of inputs, fuzzers generate test inputs at random (black-box fuzzing) or based on information about the software under test (white-box fuzzing). The latter approach uses the source code to create coverage-guided test inputs that can find defects that are opaque to many other testing approaches. Fuzz testing somewhat falls out of the classical testing setup of: unit test > integration test > system test > acceptance test, as it combines several of these steps. But it can be implemented alongside this structure, since fuzzing is best applied as a method for continuous software security testing, beginning in the early stages of software development. The CI/CD integrations that many modern fuzzing platforms offer allow developers to initiate bug fixes right away, which speeds up development dramatically.

Testing Java Applications With Feedback-Based Fuzzing

Among many security experts, feedback-based fuzzing is considered best practice for application security testing in Java. What makes this fuzzing approach so effective is that it can be largely automated. Feedback-based fuzzing approaches instrument Java applications with so-called Java agents. Java agents are custom code plugins that can be configured to feed information back to the fuzzer by using the instrumentation API to modify JVM bytecode. 

The fuzzer can then use this information to automatically generate further test cases that increase the code coverage of the current test. This allows the fuzzer to gradually refine test inputs up to a point where they become highly accurate and efficient at detecting even complex vulnerabilities that are off-limits to most security testing software. By integrating such an approach into your CI/CD, you can configure feedback-based fuzzers to run software security tests at each code change.

95% Code Coverage

XSS in JSON-Sanitizer - Testing a Popular Java Application

JSON-sanitizer is a popular Java library developed by Google and maintained by OWASP. The JSON-sanitizer’s primary purpose is to convert JSON-like content to valid JSON. Thus, its outputs should not contain substrings that might damage your scripts or even cause Cross-site scripting (XSS).

json sanitizer

Our Java testing experts used a property-based fuzzing approach with the Java fuzzer Jazzer to test this application. During this test, they would first consume a string and pass it through the JSON sanitizer. Then, the output would be analyzed for the given string. If the output would still contain it, this would mean that the application is potentially exposed to XSS.

fuzzerTestOneInput

The above fuzz target enabled our testers to generate test inputs until the result contained an invalid string which led to the XSS vulnerability. To trigger the bug, they “escaped” the first letter of an HTML tag in the JSON string. After passing it through the sanitizer, the output was the original tag, which, by nature, closes the script block. All that was left to do was to start a new script block and insert an alert to get the XSS-bug.

XXS in JSON sanitizer

Jazzer: Powerful Java Tests for Open-Source

In early 2020, Code Intelligence released Jazzer, a fuzzer for Java and other JVM-based languages. Shortly after its release, Jazzer was integrated into Google’s OSS-Fuzz, to enable Java fuzzing within Google’s powerful infrastructure. While fuzzing was already finding wide adoption in C/C++, the release of Jazzer was a huge advancement as it extended fuzzing to Java and provided the open-source community with a powerful testing tool.

"Code Intelligence's new Java Fuzzer enabled us to quickly find bugs and vulnerabilities in Java applications."

Abhishek Arya
Principal Software Engineer at Google

Abhishek_Arya_new-modified

Autofuzz - An Easy Way to Get Started With Fuzzing

Jazzer recently received an Autofuzz mode that allows users to run fuzz tests without writing test cases or harnesses. For developers who are just getting started with fuzzing, writing test harnesses is often the first big obstacle. With autofuzz, it only takes a simple command line to get your fuzzer started: 

jazzer --cp=[name of jar].jar \    --autofuzz=[name of class to fuzz]::[name of function to fuzz]

 This allows fuzzing newcomers to start fuzzing without roadblocks and gives more experienced users the tooling to speed up their testing efforts even further. Here is a quick demonstration of how autofuzz can be used to run a complete Java fuzz test in less than a minute.

 

Enough Talking, Show Me Some Code

If you want to see Java fuzzing in action, start fuzzing your own code using our fuzzing tool CI Fuzz. It comes with a JUnit integration that gives fuzz testing the look and feel of a common unit test. Try it out now, and let us know which bugs you found!

Try CI Fuzz

FAQs

Why is testing important in Java software development?

One of the most important reasons why it is important to test Java code is to ensure the software is secure. Ideally, developers test Java code early and often to identify and fix vulnerabilities before they can be exploited by malicious actors.

Testing also helps to ensure the quality of software by eliminating quality and functionality issues before a Java application is released to end users. Fixing these issues helps to prevent software crashes and other problems that can lead to lost productivity and customer dissatisfaction.

Furthermore, testing helps to keep software maintainable. By writing automated tests, developers can ensure that their software remains functional, even as it is updated and modified over time. This can save time and money in the long run by reducing the need for manual testing and bug fixing.

Finally, testing can be used to make sure that Java software is scalable and can handle high loads. This includes determining the maximum number of users a system can handle, identifying bottlenecks in a system, and measuring a system's response time given specific loads.

What is Test-Driven Development (TDD)?

Test-Driven Development (TDD) is a methodology in which tests are written for a unit of code before the code itself is written. Developers then use these tests as guidance, with the goal of ensuring that the code meets the requirements specified by the tests.

Usually, TDD follows the following steps:

  1. Write a test that defines a small piece of functionality you want to add to your code
  2. Run the test and confirm that it fails, since the functionality you want to add does not yet exist
  3. Write the minimum amount of code needed to pass the test
  4. Run the test again, and confirm that it passes.
  5. Refactor the code, if necessary, to improve its design or performance without changing its functionality
  6. Repeat the process for additional functionality

In Java code, the main benefits of TDD are that it helps to ensure that code is thoroughly tested and meets requirements. This way, developers can catch bugs early on in the development process, which can save time and resources in the long run. Additionally, TDD facilitates good software design by nudging developers to think about the functionality they want to add in small, manageable chunks.

TDD also creates a safety net for the codebase by encouraging continuous testing. The practice of TDD enables developers to catch bugs early on in the development process and also ensures that the code is maintainable, readable, and easy to understand.

How can TDD be implemented in Java?

Test-Driven Development (TDD) can be implemented in Java using the JUnit testing framework. The steps to implement TDD using JUnit are as follows:

1. Naming the test class: Use common conventions in naming the test class. If the name of the class being tested is "Student", the name of the test class should be "StudentTest". Append "Test" to the class name and use the same naming convention for test methods. For example, if there is a method called "displayStudentAddress()", the corresponding test method should be named "testDisplayStudentAddress()".

2. Packages for production code: Don't use the same package for production code and testing code. Use different source directories, "src/main/java" for production and "src/test/java" for testing.

3. Structure and Annotation: JUnit uses the @Test annotation to indicate which methods are test methods. These methods should follow the AAA pattern (Arrange, Act, Assert) where you first create objects for testing (arrange), exercise the production code (act), and then verify the results (assert).

4. Building and running the project: Build the project and run the JUnit tests using a JUnit test runner. The test runner will automatically discover and run the test methods. The tests will be marked as passed or failed based on the assertions and whether any exceptions were thrown.

5. Additional tools: You can also use tools such as JMeter for performance testing, and Mockito for creating mock objects to replace real objects in tests.

In summary, implementing Test-Driven Development (TDD) using JUnit involves following a clear naming convention, separating the production and testing code in different packages, using the @Test annotation to indicate test methods, following the AAA pattern and building the project.

What are best practices for writing unit tests in Java?

When writing unit tests in Java, it's important to follow certain best practices to ensure effectiveness and maintainability. Some key points to keep in mind include keeping tests small and focused, using test-driven development (TDD), using a test framework, mocking and stubbing, avoiding test dependencies, using a test runner, and documenting your tests. By following these best practices, you can write more effective and maintainable unit tests in Java.

What are best practices for threat modeling in Java?

Threat modeling describes the process of identifying and analyzing potential security threats to a software system. When it comes to Java, there are several best practices that can be followed to ensure that a system is designed and implemented with security in mind.

1. Understand the system: Understand the system's architecture, design, and data flow. This will help identify potential areas of vulnerability and inform the threat modeling process.

2. Identify assets: Identify the assets that need to be protected, such as user data, system resources, and network connections.

3. Use STRIDE: Use the STRIDE method to identify potential threats. This method stands for Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of privilege.

4. Use DREAD: Use the DREAD method to assess the risk of identified threats. This method stands for Damage, Reproducibility, Exploitability, Affected Users, and Discoverability.

5. Implement security controls: Implement security controls, such as input validation, access control, and encryption, to mitigate the identified threats.

6. Regularly review and update: Regularly review and update the threat model as the system evolves, new threats are identified, and security controls are added or updated.

7. Use security tools: Use security tools such as static code analysis, penetration testing, and runtime application self-protection (RASP) to identify and address vulnerabilities in the code.

8. Test the system: Test the system to ensure that it is secure and that security controls are working as intended.

In summary, to implement threat modeling in Java, you should understand the system, identify assets, use threat modeling methods such as STRIDE and DREAD, implement security controls, regularly review and update the threat model, use security tools, and test the system. These practices will help ensure that the system is designed and implemented with security in mind and that potential threats are identified and mitigated.