Comparison and analysis of five different

open source bug tracking systems

SUBMITTED BY:Name RAHUL ROY

Prakash Tiwari

Regd. NO-11005675
Rollno A59
CSE526

GUIDED BY:
Sudhanshu

TABLE OF CONTENT:

Introduction
Bug Tracking System
Classification Criteria
Introduction to BugZilla
Introduction to Mantis
Introduction to Jira
Classification based on platform
Classification based on support
Classification based on size and usage
Classification based on reporting
Classification based on tracking
Classification based on other features
Conclusion

INTRODUCTION:
As software projects become increasingly large and complex, it becomes
more difficult to properly verify code before shipping. Maintenance activities
account for over two thirds of the life cycle cost of a software system, summing up
to a total of $70 billion per year in the United States. Software maintenance is
critical to software dependability, and defect reporting is critical to modern
software maintenance.
Large software development projects require a bug tracking system to
manage bug reports and developers who work on fixing them. A ubiquitous
example of such a system is Bugzilla, an open-source system first introduced in the
development of the Mozilla web browser, but now used in numerous other
projects.
Bug tracking systems are particularly important in open source software
development, where the team members can be dispersed around the world. Thus,
the bug tracking system is used not only to keep track of problem reports and
feature requests, but also to coordinate work among the developers.
Many bug-tracking systems, such as those used by most open source
software projects, allow users to enter bug reports directly. Other systems are used
only internally in a company or organization doing software development.
Typically bug tracking systems are integrated with other software project
management applications. Having a bug tracking system is extremely valuable in
software development, and they are used extensively by companies developing
software products.
In proprietary software, the fixing of bug takes place in a phased manner
because each phase has separate team responsible for the work being done in that
phase. While in the open source project, anyone can pick up the bug and start
working on that. Dependency may become major issue in calculating the time for
fixing of the bug. If the bug is identified before the implementation phase then it
can be very well resolved by development team or owner of the module. But once
the software is released and implemented, it becomes very difficult to fix the bug.
It has become a cumbersome job for the moderator to find a suitable
developer who can fix a particular bug because the description of the reported bug
may not generally provide complete information. Many organizations generally

rely on email with or without attachment/feedback based bug reporting system and
these bugs will be maintained in spreadsheet or in any document editing software.
It again becomes very difficult for the owner or developer to track the bug and its
progress of fixing.

Bug Tracking System

The bug reporting/tracking system should provide an interactive web based
platform for bug reporting and tracking the progress. The system may involve a
generic process or specific schedule of bug reporting process. The generic process
of the reporting a bug may generally contain the following information:
Title: Title of the bug.
Description: Detailed description of the bug including what, where, why,
how and when the bug occurs. Actual message that appears during the
operation may be included with the actual set of input and expected output.
Version: Specify the version of the project.
Component: Specific component need to be specified.
Screenshot/Attachment: Corresponding screenshot can also be uploaded as a
.jpg or .gif file by capturing the actual operation/output/message.
Priority: Priority may be assigned for its urgency.
Severity: Specify frequent occurrence and its impact in the system.
Status: Current status of the bug (new, opened, confirmed, closed, etc.).
Created by: Name of the person or id already registered with the system who
is reporting the bug.
Assigned to: The bug reporter may also assign bug to specific person, if one
knows about a particular person who can solve this problem otherwise
assigned by the moderator.
Revision History: If the bug is earlier reported, show the historical changes.

 Estimated time: The estimated time may be specified, generally used in case
of closed team environment not in the open source environment.
Comments: Any other information that will be helpful in identifying the bug.

Bug Life Cycle

When the bug is first reported, its status must be set to Unconfirmed as in the
case of bugzilla. After its first review, the status can be updated to New or
Assigned based on the previous bug history databases. Once the assignee has
corrected the bug, he or she can update its status as Resolved by specifying how
it was resolved i.e. fixed, invalid (not a bug), duplicate, won't fix, and the
(in) famous works for me. A resolved bug, will not be closed until someone
else will confirm it out or it will be confirmed by the moderator. Once it is

confirmed, the bug status becomes verified otherwise the status must be set to
Reopen. It remains in Verified state until it is incorporated in the release
version of the product and its status will be updated to Closed. The closed bug
can be reopened with its status as New or Unconfirmed. There might be many
bugs that are currently being fixed and some of bugs that are waiting for resolution
set to be closed.

Classification Criteria
The bug tracking system varies from general purpose to the specific purpose.
The generic system will have many features while the specific purpose systems
may have limited features. A suitable tool can be selected based on users
requirement. The requirement generally varies from the platform, support, size,
reporting, tracking and other features of the project. These broad criteria can be
further classified or categorized in three or more sub-categories.

Platform refers to license policy, architecture, operating system, web server,

back-end database, programming language and clients. Support refers to language,
multiple projects, web interface, database, email notification and localization. Size
and usage refer to the size of the project, number of downloads, number of releases
and number of clients/usage. Reporting refers to form based, email, attachments,
web and feedback. Tracking refers to timely update status of the bug, graphs for
number of pending bugs, assigned, fixed over a period of time in each category and
search facility for new as well as existing bugs to know about their status. The

other feature refers to any other feature that is not a part of the above mentioned
categories.

FIVE OPEN SOURCE BUG TRACKING SYSTEMS ARE:

BUGZILLA

MANTIS
JIRA
BUGTRACKER.NET
FOSSIL

Bugzilla
INTRODUCTION:
Bugzilla is a Web-based general-purpose bug tracker tool originally developed
and used by the Mozilla project. Released as open source software by Netscape
Communications in 1998, Bugzilla has been adopted by a variety of organizations
for use as a defect tracker for both free software and proprietary products.
Bugzilla is licensed under the Mozilla Public License. Bugzilla was
originally written by Terry Weissman in 1998 for the nascent Mozilla.org project,
as an open source application to replace the in-house system then in use at
Netscape Communications for tracking defects in the Netscape Communicator
suite. Originally written in Tcl, Terry decided to port Bugzilla to Perl before its
release as part of Netscape's early open source code drops, with the hopes that
more people would be able to contribute to it as Perl seemed to be a more popular
language at the time.
Bugzilla 2.0 was the result of that port to Perl, and the first version released
to the public via anonymous CVS. In April 2000, Weissman handed off control of
the Bugzilla project to Tara. Hernandez. Under Tara's leadership, some of the
regular contributors were coerced into taking more responsibility, and Bugzilla
development became more community-driven.

In July 2001, facing distraction from her other responsibilities in Netscape,

Tara handed off control to Dave Miller, who is still in charge as of June 2007.
Bugzilla 3.0 was released on May 10th 2007 and brought refreshed UI, XML-RPC
Interface, custom fields and resolutions, mod Perl support, shared saved searches,
improved UTF8 support and others.

REQUIREMENTS:
Bugzilla's system requirements include:
A compatible database management system;
A suitable release of Perl 5;
An assortment of Perl modules;
A compatible web server;
A suitable mail transfer agent or any SMTP server.
Currently supported database systems are MySQL and PostgreSQL. Bugzilla is
usually installed on Linux and runs using the Apache HTTP Server, but Microsoft
Internet Information Services or any web server that supports CGI can be used.
Bugzilla's installation process is command line driven and runs through a series of
stages where system requirements and software capabilities are checked.

Anatomy of a Bug:
The core of Bugzilla is the screen which displays a particular bug. Bug 1 on
Landfill
(http://landfill.bugzilla.org/bugzilla-tip/show_bug.cgi?id=1) is a good example.
The labels for most fields are hyperlinks; clicking them will take you to contextsensitive help on that particular field. Fields marked * may not be present on every
installation of Bugzilla.
1. Product and Component: Bugs are divided up by Product and Component, with
a Product having one or more Components in it.
For example, bugzilla.mozilla.orgs "Bugzilla" Product is composed of several
Components:
Administration: Administration of a Bugzilla installation.
Bugzilla-General: Anything that doesnt fit in the other components, or spans
multiple Components.
Creating/Changing Bugs: Creating, changing, and viewing bugs.
Documentation: The Bugzilla documentation, including The Bugzilla Guide.
Email: Anything to do with email sent by Bugzilla.
Installation: The installation process of Bugzilla.
Query/Bug list: Anything to do with searching for bugs and viewing the bug lists.
Reporting/Charting: Getting reports from Bugzilla.
User Accounts: Anything about managing a user account from the users
perspective.
Saved queries, creating accounts, changing User Interface: General issues having
to do with the user interface cosmetics (not functionality) including cosmetic
issues, HTML
2. Status and Resolution: These define exactly what state the bug is in - from not
even
being confirmed as a bug, through to being fixed and the fix confirmed by Quality
Assurance.
3. Assigned To: The person responsible for fixing the bug.
4. *QA Contact: The person responsible for quality assurance on this bug.
5. *URL: A URL associated with the bug, if any.
6. Summary: A one-sentence summary of the problem.
7. *Status Whiteboard: (a.k.a. Whiteboard) A free-form text area for adding short
notes
and tags to a bug.

8. *Keywords: The administrator can define keywords which you can use to tag
and
Categories bugs - e.g. The Mozilla Project has keywords like crash and regression.
9. Platform and OS: These indicate the computing environment where the bug was
found.
10. Version: The "Version" field is usually used for versions of a product which
have
been released, and is set to indicate which versions of a Component have the
particular problem
the bug report is about.
11. Priority: The bug assignee uses this field to prioritize his or her bugs. Its a
good idea
not to change this on other peoples bugs.
12. Severity: This indicates how severe the problem is - from blocker ("application
unusable") to trivial ("minor cosmetic issue"). You can also use this field to
indicate whether a
bug is an enhancement request.
13. *Target: (a.k.a. Target Milestone) A future version by which the bug is to be
fixed.
14. Reporter: The person who filed the bug.
15. CC list: A list of people who get mail when the bug changes.
16. *Time Tracking: This form can be used for time tracking. To use this feature,
you
have to be blessed group membership specified by the timetrackinggroup
parameter.
Orig. Est.: This field shows the original estimated time.
Current Est.: This field shows the current estimated time. This number is
calculated from Hours Worked and Hours Left.
Hours Worked: This field shows the number of hours worked.
Hours Left: This field shows the Current Est. - Hours Worked. This value +
Hours Worked will become the new Current Est.
%Complete: This field shows what percentage of the task is complete.
Gain: This field shows the number of hours that the bug is ahead of the Orig.
Est..
Deadline: This field shows the deadline for this bug.
17. Attachments: You can attach files (e.g. testcases or patches) to bugs. If there
are any attachments, they are listed in this section. Attachments are normally stored
in the Bugzilla database, unless they are marked as Big Files, which are stored
directly on disk.

18. *Dependencies: If this bug cannot be fixed unless other bugs are fixed
(depends on), or this bug stops other bugs being fixed (blocks), their numbers are
recorded here.
19. *Votes: Whether this bug has any votes.
20. Additional Comments: You can add your two cents to the bug discussion here,
if you have something worthwhile to say.

MANTIS:
INTODUCTION:
Mantis Bug Tracker is a free web-based bugtracking system. It is written in
the PHP
scripting language and works with MySQL, MS SQL, and PostgreSQL
databases and a
webserver. Mantis Bug Tracker claims support for Microsoft Windows, Mac
OS, OS/2, and a variety of Unix operating systems. It is released under the
terms of the GNU General Public License.
Mantis Bug Tracker is also distributed with most Linux distributions.
The goals of this project are to produce and maintain a lightweight, simple
bugtracking system. Additions of complexity/features are modular so that
users can be shielded from unwanted clutter. Thus, much of the package has
a simple version of a feature along with a more fully developed version.
In the 'core' package, the aim is to have the most important, most
used, most time saving portions of a bugtracking system. The product is
designed to be easily modifiable, customizable, and upgradeable. Anyone
with intermediate PHP and MySQL experience should be able to customize
Mantis to suit their needs.

Anatomy of a bug:
The bugs are listed in a table and the attributes are listed in the following order:
Priority;
Id;
Number of bugnotes;
Category;
Severity;
Status;
Last updated;
Summary.
Each (except for number of bugnotes) can be clicked on to sort by that column.
Clicking again will reverse the direction of the sort. The default is to sort by last
modification time, where the last modified bug appears at the top.

The bug id is a link that leads to a more detailed report about the bug. Depending
on what you have set in your Account Preferences you will be sent to the simple or
advanced view. The user can also add bugnotes here.
The number in the bugnote count column will be bold if a bugnote has been
added in the specified time frame. The addition of a bugnote will make the bugnote
link of the bug appear in the unvisited state.
The text in the "Severity" column will be bold if the severity is major, crash,
or block and the bug not resolved.
Types of severities:

block - prevents further work/progress from being made

crash - crashes the application or OS
major - major bug; there is no workaround
minor - not trivial, but there is a workaround
tweak - needs tweaking
text - error in the text
trivial - extremely minor issue
feature - issue does not describe a problem

JIRA
INTRODUCTION:
JIRA is a bug tracking, issue tracking, and project management application
developed to by Atlassian Software Systems is an Australian software company
specializing in issue tracking and collaboration software. JIRA has been designed
with a focus on task achievement, is instantly usable and is flexible to work with.
JIRA is available in three editions to accommodate your bug tracking, issue
tracking, and project management needs:
JIRA Standard
JIRA Professional
JIRA Enterprise

The main characteristics of JIRA are:

Manage bugs, features, tasks, improvements or any issue
A clean and powerful user interface that is easy to understand for both
business
and technical users
Map your business processes to custom workflows
Track attachments, changes, components and versions
Full text searching and powerful filtering (customizable, savable, shareable
and
subscribe able!)
Customizable dashboards and real-time statistics
Enterprise previsioning and security
Easily extended to and integrated with other systems (including email, RSS,
Excel, XML and source control)
Highly configurable notification options
Runs on almost any hardware, OS and database platform
Web service enabled for programmatic control (SOAP, XML-RPC and
REST
interfaces)

Anatomy of a bug:
JIRA tracks issues, which can be software bugs, feature requests, project
tasks, helpdesk tickets, leave request forms, or any other tasks the user want to
track.
The numbered fields as follows:
Key - a unique identifier for this issue.
Type - the type of issue that is reported by the user.
Status - the stage the issue is currently at in its lifecycle.
Resolution - a record of the issue's resolution.
Priority - the importance of the issue in relation to other issues.
Assignee - the person to whom the issue is currently assigned.
Reporter - the person who entered the issue into the system.
Project - the 'parent' project to which the issue belongs.
Component(s) (if applicable) project component(s) to which this issue
relates.
Affects Version(s) (if applicable) - project version(s) for which the issue is
(or was) manifesting.
Fix Version(s) (if applicable) - project version(s) for which the issue was (or
will be) fixed.
Summary - a brief one-line summary of the issue.
Environment (if applicable) - the hardware or software environment to
which the issue relates.
Description - a detailed description of the issue.
Comments - added by people who are working on the issue.
Additionally, if the administrator has enabled 'Time-Tracking', the following
fields will appear above the 'Description' field:
Original Estimate - the total amount of time required to resolve the issue, as
estimated when the issue was created. (This field cannot be modified once
work has been logged on the issue.)
Remaining Estimate - the remaining amount of time required to resolve the
issue. This field only appears when work has been logged on the issue.
Time Spent - the sum of all the individual work logs for this issue. This field
only appears when work has been logged on the issue.

Classification Based on Platform Characteristics :

The comparison criteria based on platform characteristics. In this table, most
of these tools are available in open source environment. Mantis is closed source

while its free version is available for download and use. Some of the tools are
available as a licensed version for which support can be asked from the owner.
Paid version of Jira comes in three different variants as Standard, Professional and
Enterprise based on the size of the project and support. Most of these tools work in
web based environment while their early versions were available as client/server
based environment.
Trac is developed on wiki based architecture, i.e. anyone can see bugs, fix
bugs and edit any part of it. Operating environment for most of the projects are
cross-platform, i.e. they can be installed on any machine. BugZilla and GNats are
available on Linux while BugTracker.Net is available on windows environment.
The clients are browser based, so they are independent of the platform. For web
server, most of the tools required apache/tomcat except BugTracker.Net which
requires MS-IIS. Most of these tools support MySQL as a backend database except
BugTracker.net which requires MS-SQL Server and Fossil requires SQLLite.
These tools may vary on the basis of programming language used in coding. Tools
are developed in C, Python, and PHP, Perl, Java and C #.Net programming
languages.
Tools/Platfor
m
Platform

BugZilla

Jira

Mantis

Open
source/
Free/
Proprietary

Proprietary/
Free for
Non
commercial

Free/
Paid

System
Architecture

Client
Server/
Web Based

Client
Server/
Web Based

Server
Operating
System

Linux

CrossPlatform

Web
Server

Apache,
MS-IIS or
Server
capable to
run CGI
MySQL,
Oracle and
PostgreSQL

Backend
Database

BugTrack.Ne
t
Open
Source

Fossil

Web Based/
WAP

Web Based

Distributed
Web
Based

Windows

Linux/
Mac OS/
Windows

Apache
Tomcat

Windows,
Linux, Mac
OS
OS/2 and
others
Apache,
MS-IIS

MS-IIS

Apache,
MS-IIS

Mostly
supports all
RDBMS

MySQL,
Oracle and
PostgreSQL

MySQL
Server

SQLLite

Open
Source/
Free

Programmin
g
Language
Client (web
Browser)

TCL/perl

Java

PHP

ASP.NET

Anyone

Anyone

Anyone

Anyone

Anyone

Classification Based on Support Characteristics:

The general characteristics for support features are Language, Web Interface,
Maintenance support, Email notification and Localization. Most of the tools
support multiple languages based on Unicode system. Only Fossil and GNats do
not have the support of multiple language features. Again in the same line, most of
the tools support web based interface. People are more fascinated to use browser
based interface because it is independent of operating system environment. Fossil
does not have the complete web based interface.
Maintenance support is also available for free of cost or with a paid service
with nominal charges. Whenever, a new bug or update is reported, an email will be
automatically sent to all users who are in the registered mailing list with the
project. The email will also be sent to all registered users even when there is a
small update. Bug localization, i.e. finding the most relevant part of source file
hierarchy of software in the bug repository database is another important feature
provided by BugZilla, Trac and Mantis while others does not have.
Tools/Support
Language
Support
Web Interface
Maintenance
Support
Email
Notification
Bug
Localization

BugZilla
Yes

Jira
Yes

Mantis
Yes

BugTracker.Net Fossil
Yes
No

Yes
Yes

Yes
Yes

Yes
Yes

Yes
Yes

No
Yes

Yes

Yes

Yes

Yes

No

Yes

No

Yes

No

No

Classification Based on Size and Usage Characteristics:

In this category, size of the project, support of multiple projects, number of

downloads number of releases and clients per usage are the important parameters
for which the statistics can be collected over different time period that may be from
the date of first release to the recent date of release.
Tools/Release Bugzilla
Status
First Release 1998
Latest
2011
Release

Jira

Mantis

BugTracker.Net Fossil

2004
2010

2000
2010

2002
2011

2006
2011

Year Wise Number of Releases:

Tools/Year
2011
2010
2009
2008
2007
2006
2005
2004

BugZilla
4
14
15
10
7
8
9
3

Jira
1
2
1
1
5
3
4
1

Mantis
5
6
9
8
11
10
6

BugTracker.Net
14
28
36
15
-

Classification Based on Reporting Characteristics :

Tools/Reporting
Form Based
Email
Attachments
Web(Online)

BugZIlla
Yes
Yes
Yes
Yes

Jira
Yes
Yes
Yes
Yes

Mantis
Yes
Yes
Yes
Yes

BugTracker.Net
Yes
Yes
Yes
Yes

Fossil
Yes
No
No
Yes

Feedback

Yes

Yes

Yes

Yes

Yes

A well designed reporting feature will attract users in submitting the bug
effectively so the project will not miss any of the failure report instance. For
example, in the Eclipse project, the submission of bug reports can be done
automatically as well as its usage.
The user will click only once in the checkbox and later one can click on a
button to submit the bug to the project that will get updated in the system.
Therefore, if the bug reporting system contains all these methods, then it becomes
very efficient to know about the bug because user will not hesitate in submitting
the bugs if he follows a click and proceed approach. BugZilla, Jira, Mantis and
BugTraccker.Net have almost every reporting facility but Trac and Fossil are
lagging behind in attachments and email based reporting.

Classification Based on Tracking Characteristics:

The tracking feature will contain subcategories likely timely updates,
graphical display and search facility. Most of the systems have these functionalities
as shown. This can be further categorized to one more level. Whenever the new
updates are received, the registered members will receive an email regarding the
updates as well as new release of the system.
The graphical and charting facility is one of the very important features that
tell user about the number of pending bugs, number of open bugs, number of
closed bugs, time taken by each bugs in the form of interactive charts and
estimated time to resolve the bug. To track the bug, the tools have the facility to
search for a bug that is already submitted in the system. This will be helpful in
identification of duplicate bugs. Current days product must have all these facility
with dynamic graphical and charting options. The search facility is also one of the
important considerations. Generally, the systems have full text search facility on
title, description and summary of the bug report.
Tools/Tracking BugZilla
Timely
Yes
updates

Jira
Yes

Mantis
Yes

BugTracker.net Fossil
Yes
Yes

Graphical
Display/
Reporting

Yes

Yes

Yes

Yes

Yes

Search
Facility

Yes

Yes

Yes

Yes

Yes

Classification Based on Other Features:

Any other feature may incorporate many more features that are very
important but may not have one special category. These are bug localization,
change prediction, browsing of source code repository, version control, and
Subversion facility. Some of the systems have the capability of these features but
they are not mature.
Overall the above mentioned subcategories can be further sub divided into
specific subgroups. It will enable to provide a better picture of the bug reporting
and tracking system. It becomes very helpful in choosing a tool that will meet the
user specified criteria and need.

Conclusion:
A good automated bug-tracking tool should streamline the process of
reporting, managing and fixing issues. In this paper, authors reviewed some of the
bug tracking systems and then present a comparative analysis of different bug
tracking system based on different classification criteria. It will help the developers
to choose bug tracking tool as per their requirement and constraint.
We have also presented a theoretical framework for developing bug tracking
and reliability assessment system (BTRAS) based on different consideration. The
proposed BTRAS will assess the reliability of software and give information about
bug complexity, bug distribution to developer apart from reporting and tracking.