A GSP Bulletin provides essential information to Ground Service Providers on

matters of an ISAGO Audit, ISAGO Registration or Station Accreditation or

the requirements of the ISAGO Audit Program. It is published by IATA as
necessary, reviewed periodically and remains in force until revoked.

2020/6 AUDITING EFFECTIVENESS IN ISAGO

ISAGO Standards and Recommended Practices (GOSARPs) have traditionally been audited for
conformity on the basis of “documented” and “implemented”. This method proved adequate for simple
and straightforward processes and procedures. Today’s complex and diverse operations mean that the
processes and procedures can also be complex and diverse and the means of achieving conformity
shouldn’t be based only on a review of the documentation and records of implementation. This is
particularly true for management systems.

For ISAGO audits conducted against Edition 9 of the ISAGO Standards Manual (GOSM), the assessment
of effectiveness will be performed on a trial basis in addition to the existing assessment. The trial will
initially be limited to only one GOSARP, ORM 1.3.4. The auditor will seek your assistance and you will be
expected to provide evidence to demonstrate the effectiveness criteria are met. The outcome of the
assessment of effectiveness will not influence the assessment of conformity of ORM 1.3.4.

The auditor will record the result of the assessment in the audit software. IATA will collate and evaluate
the results on a regular basis and will assess the outcome of the tail when enough data is obtained.

The Appendix to this GSP Bulletin explains the assessment in more detail. Your audit team will also
provide more information during the audit.

We thank you in advance for your cooperation in this trial.

Approved by: Paul Fleming, Head, Ground Ops Audits

Date: 23 June 2020
GSP Bulletins currently in force:

2018/7 15 June 2018, Corrective Action Records

2018/9 16 August 2018, Guidelines for Auditees - ISAGO Audit Software User Manual & ISAGO Audit Follow-
Up Activities

2019/1 2 January 2019, GOPM Edition 2

2019/4 17 May 2019, Temporary Revision 1 to GOSM Edition 8

2019/5 10 October 2019, ISAGO Registration/Station Accreditation Press Releases and Use of IATA Brand and
ISAGO Logos

2019/7 20 December 2019, Audit Allocation 2020

2020/1 15 January 2020, ISAGO Standards Manual (GOSM) Edition 9

2020/4 25 March 2020, ISAGO Program Manual (GOPM) Temporary Revision (TR) 1, Coronavirus (COVID-19)
Measures for ISAGO Audits

2020/5 31 March 2020, Application of Extenuating Circumstances to the Impact of COVID-19

2020/6 23 June 2020, Auditing Effectiveness in ISAGO

Appendix to GSP Bulletin 2020/6 Auditing Effectiveness in ISAGO

ISAGO Standards and Recommended Practices (GOSARPs) have traditionally been audited for conformity on the
basis of “documented” and “implemented”. This method proved adequate for simple and straightforward processes
and procedures. Today’s complex and diverse operations mean that the processes and procedures can also be
complex and diverse and the means of achieving conformity cannot be based only on a review of the documentation
and records of implementation. This is particularly true for management systems.

Conformity with a GOSARP will still be based on documentation and implementation but there also has to be a more
in-depth approach that assesses whether the implementation of the processes and procedures achieve the desired
outcome. That they are effective. If the requirements of the GOSARP relate to a management or operational
objective or outcome, the assessment should also verify that these requirements are achieved.

The term “effective” may be defined as: the desired outcome is achieved when a GOSARP is assessed as
documented and implemented and additionally, the defined suitability and effectiveness criteria as specified in the
Assessment Tool are fulfilled. This definition clearly indicates that the assessment of document and implemented
remains unchanged but also that two new criteria are introduced – suitability and effectiveness. For the purpose of
the initial implementation we will concentrate on effectiveness. The definition also informs that an “Assessment
Tool” is to be used.

The effectiveness criteria in the Assessment Tool are not currently included as or part of a GOSARP. They are,
however, elements or indicators that could be associated with the GOSARP requirements that, when implemented,
should achieve the desired outcome. The GOSARP itself therefore must be amended to include the criteria to
introduce the assessment of effectiveness properly. This is work in progress and will be influenced by the initial
assessments of effectiveness.

The assessment of effectiveness is indicated by the symbol [Eff] and is initially applied to one GOSARP in the ISAGO
Standards Manual (GOSM) Edition 9.

ORM 1.3.4 The provider shall have a safety risk assessment and mitigation program that includes processes
implemented and integrated throughout the organization to ensure:
(i) Hazards are analyzed to determine corresponding safety risks to ground operations.
(ii) Safety risks are assessed to determine the requirement for risk mitigation action(s).
(iii) When required, risk mitigation actions are developed and implemented in operations. [Eff] (GM)

In assessing the effectiveness criteria, the auditor will need to examine records, reports or some other documented
information looking for evidence that demonstrates the requirements of the GOSARP have been implemented
effectively. The auditor will therefore request you to provide specific information.

In the case of ORM 1.3.4 and the area of Safety Risk Management it should be possible for the auditor to follow a
documented trail of the safety risk management process. This trail could start with the analysis of a safety report
through to a conclusion (either that no action need be taken or a recommendation for mitigation). The auditor will
look for evidence of a safety risk management process that produces results. The results at each stage of the
process should be clearly recorded and conclude in reasonable logical outcomes. The auditor may examine two or
three different examples to get a full picture of how well the safety risk management process is working. The auditor
will also look for the use of tools that are based on the guidelines published by an aviation authority or in a software
package. There are many software packages available for risk management.
Using the Assessment Tool

The Assessment Tool takes the form of a series of Auditor Actions associated with the GOSARP, as shown below.
You will see that suitability and effectiveness criteria are specified together.

Effectiveness:

 i) The safety risk assessment and mitigation program is suitable for the size, complexity and nature of the
ground operations.
 ii) All relevant hazards are analyzed for corresponding safety risks.
 iii) Safety risks are expressed as likelihood of the occurrence and the severity of the consequence of the
occurrence.
 iv) A matrix quantifies safety risk tolerability to ensure standardization and consistency in the risk assessment
process which is based on clear criteria.
 v) Risk register(s) across the organization capture risk assessment information, risk mitigation (control) and
monitoring actions.
 vi) The risk mitigation (control) actions include timelines, allocation of responsibilities and risk control
strategies such as hazard elimination, risk avoidance, risk acceptance, risk mitigation.
 vii) Mitigation (control) actions are implemented to reduce the risk to a level of “as low as reasonably
practical”.
 viii) Identified risks and mitigation actions are regularly reviewed for accuracy and relevance.
 ix) Effectiveness of risk mitigation (control) actions are monitored at least yearly to include auditing in
accordance with ORM 1.4.1.
 x) Personnel performing risk assessments are appropriately trained in accordance with ORM 4.3.1.
 xi) The program takes into consideration any area of the organization where there is a potential for hazards
that could affect ground operations and the services provided.
 xii) The program has some form of central coordination to ensure all existing or potential hazards that have
been identified as relevant are subjected to risk assessment and, if applicable, mitigation.

The auditor will check the box against each criterion in the audit software if satisfied that your implementation of
ORM 1.3.4 is effective in each area described in the criterion.

Results

Even though your SMS is functional and meets the requirements of the SMS standard ORM 1.1.3, the auditor’s
assessment may not confirm that all safety risk management functions are effective. In this case, the auditor will
discuss the assessment result with you. The auditor will also provide a short report in the audit software that
summarizes the assessment in the report. Remember, the assessment of effectiveness is currently a trial and not
considered in the assessment of conformity of ORM 1.3.4.

Further Reading

You can find more information on auditing and assessing effectiveness in the IOSA Audit Handbook, which you can
download here. The European Aviation Safety Agency (EASA) also has useful guidance on the use of an assessment
tool, which you can download here. There are also many instructional videos online that you can view, with
appropriate caution.

- END -