@Q Shell Oil Products US Martner Refinery .0. Box 711, Martinez, C8 98853-0071 io hond delivery o CCC Hazardous Meterals Dison Aandal Sawyer Chief Environmental ath and Hatardous Material fer (Contra Costa Haradous teri Programs 4585 Pacheco Boulevard Suite 100 Re: 30:ay Repor for 2/19/16 Incident (Partial Power Outage and Fring] Dear me, Soyer, ‘ached the 30-day report regarding the arngincdent which occured at Shll Mariner Refinery on December 9, 2016. This epart macs the raquremens nthe “Matrdous Matralsnint Notion Paley” ated anuary 22,2016 Ifyou have any questions, please contact meat 925-313.3079 or va [email protected] sincerely, a pup Process Safety Manager ‘tachments: — 30day Follow Notation Report Form Investigation Report Michael Dassey, CCCHNDATTACHMENT € For CCHS Use Only: 0-DAY FOLLOW-UP NOTIFICATION REPORT FORM. CONTRA COSTA HEALTH SERVICES Received By: Date Received: INSTRUCTIONS: A hardcopy and an slectonic copy of this par isto | ineident Number be submit fal oe 2 an inn owe aves BY CCHS. | Gop To See Attachment C-1 for suggestions regarding the type of information o be incl inthe opr Attach ational shea ncoeseay. This orm is |_EVR Clasefeation Love tobe used for update reports afer tho initial 30-day report has been submited. Forward the completed form to: ATTENTION: Randall L. Sawyer (Chief Environmental Health and Hazardous Materials Otter Conia Costa Hazardous Materials Programs 4585 Pacheco Boulevard, Suite 100 Martinez, CA. 94553 INCIDENT DATE: December 19, 2016 INCIDENT TIME: 1:15 PM (approsimately) FACILITY: ‘Shell Martinez Refinery PERSON TO CONTACT FOR ADDITIONAL INFORMATION HaNawyen 9253313-3079 PROVIDE. ANY ADDITIONAL INFORMATION THAT WAS NOT INCLUDED IN THE 72- HOUR REPORT WHEN THE 72-HOUR REPORT WAS SUBMITTED, INCLUDING MATERIAL RELEASED AND ESTIMATED OR KNOWN QUANTITIES, COMMUNITY IMPACT, INJURIES, BTC. |. INCIDENT INVESTIGATION RESULTS Is the investigation ofthe incident compte a this time? Yes ___No I the answer is no, when do you expect completion of the Investigation? [tie answer is ys, complete the following: SUMMARIZE INVESTIGATION RPSULTS BELOW OR ATTACH COPY OF REPORT: SEE ATTACHED INVESTIGATION REPORT January 22, 2016 Page t30-DAY REPORT, PAGE 2 INCIDENT DATE: 12/19/16 FACILITY: I Martinez Refinery SUMMARIZE. PREVENTATIVE MEASURES TO BE TAKEN TO PREVENT RECURRENCE INCLUDING MILESTONE AND COMPLETION DATES FOR IMPLEMENTATION: ‘Recommendations Extimated Completion Date T. pine Herc ob Say Analysis (SA) empleo clade vege fo eval the need tor ations elect sa we eubespooane of breaker contra and potetiverlaying in main substations (Substation 1,2, | 8942017 3), 2 Ans abel For potest relays on Pane in So 1 Labels wil identify the coreponding reser tht wl rp up atvation oft rotective ansnois hy. 3. sal infra abel for alarms associ with reer son alarm patel (Pel 6) Sub ansianin 4, Revow intent invesigation lerings wih ll applizableeletcal finer 02017 STATE AND DESCRIBE THE ROOT-CAUSE(S) OF THE INCIDENT: SEE ATTACHED INVESTIGATION REPORT January 22, 2016 Page 2Level 2 TOP/CL SMR2016-007 Investigation Report Loss of Power to Substations 1203 & 1206 Date & Time the Incident Began: 12-19-2016 (~13:15) Tnvestigation began on 12-19-2016 (~13:15) with the colletion of data based na lock (itary time), unl ed N roximat 1.0 Summary of Event At approximately 13:15, on December 19, 2016, Substation (Sub) 1 Bus Section G main breaker (1700/2) ‘tipped causing loss of power to two I2kV distribution substations, Sub 1203 and Sub 1206. Sub 1 is one of Shell's three main electrical substations that feed the refinery. This resulted in loss of power to multiple units in LOP, OPCEN, Utilities and Logistics Vine Hill area. Subsequently, the affected units were shut down due to ‘oss of power, ‘The Sub | Bus G main breaker (1700/2) was manually reclosed at 13:17, restoring power to Sub 1203 and Sub 1206, 2.0 Impact ‘The multiple unit's shutdown caused flaring at the LOP and FXG flares, which resulted in multiple odor and noise complaints from the community. ‘Asa precaution, a Community Warning Level (CW) 2 was activated. An incident that triggers an activation of CWL 2 is considered a Major Chemical Accident or Release (MCAR) per the CCHS Guidance Document Definition. 3.0 Background and Process Description Martinez has three main substations. Each main substation receives power from two-115kV PG&E transmission Tines and steps down the voltage to 12V for distribution to unit substations. Shell purchased SubI from PG&E. inthe early 1980's. The substation was designed per PORE standards a the time, ‘The other two Shell substations, Sub 2 and Sub 3, were designed to Shell standards. Under normal configuration and operation, both 115kV PG&E lines into Sub 1 ae in-service and provide power through two transformers (1106 and T107). A steam turbine generator (STG) also distributes and provides power to Sub 1 In this configuration, the F/G Tie breaker remains inthe open positon, ‘When the F/G Tie breaker is closed, it connects Bus Section G and Bus Section F which allows a path for power tobe distributed between the two bis sections. After the addition of the STG and associated power distribution system in 1995, the F/G Tie breaker remains inthe open position during normal operation beeause the breaker is under-rated for the avsilable fault curent from both transformers (T106 and T107) and the STG. Please reference associated Eletrical Diagrams in Attachment A for more detail ‘Several times during the year, Subl is switched toa one transformer service (using T106 oF T107) in order for Shell o perform substation maintenance or atthe request of PGA&E for switching. When Sub 1 is being supplied power by one transformer, the FIG Tie breaker is manually closed in order to allow for a redundant path of power to and from Bus Section F and Bus Section G.Level 2 TOP/CL SMR2016-007 Subl is unique from other main substations in that itis comprised of outdoor breakers (ISKV and 115KV) and ‘open buses. The indoor contol building for Sub | contains separate panels for breaker contol and relaying, The TG Tie breaker conteol and relays are located in the Sub I control building, while the wires are run extemal from Sub | control building tothe outdoor F/G Tie breaker. ‘This substation was originally designed and owned bby PGA&E but was purchased by Shell in the 1980's. Unlike Sub 1, Shell's other major substations (Sub 2 and Sub 3) were designed to Shell specifications. They are comprised of power distribution center with an indoor lineup of 12KV switchgear, which contains the breakers, relays and associated control witing inside the control building 4.0 Narrative Sequence of Events/Timeline (On Tuesday December 13, 2016, PGAE requested a clearance in order to operate their tiebreaker. This requited Shell to switch to one tansformer service at Sub 1, Sub 2, and Sub 3. During the switching activity on December 13%, it was discovered tha the FIG Tie breaker would not close wien the breaker control switch, located in Sub'l contol building, was moved to the “closed” position. Upon activating the FIG Tie breaker control switch in the elosed position, an alarm it up onthe pane! indicating “I2KV Bus Section Switch F/G Tie “Trip”, The PG&E switching was completed successfully with the FIG Tie breaker in the open position, Shel, personnel determined to complete the switching activity while the F/G Tie breaker remained in open position ‘because operating off ofa one tansformer service was going tobe fora short duration (couple of hours). After the switching was completed, on December 138, 2016, the electrical group sill needed to determine what initiated the alam and why the FIG Tie breaker remained open. On Monday December 19%, 2016, troubleshooting on the F/G Tie breaker was planned to occur. Ths date was chosen to avoid the previous ‘week's inclement weather At 10:28, on December 19" the Blectrcal Staff’ member (E.S.) received a safety permit from the Uilties ‘department o inspect the witing and components ofthe F/G Tie breaker and the associated “12kV Bus Sectionalizing F&G” panel Panel 2) as part of his initial troubleshooting. Upon inspecting the outdoor FIG Tie breaker internals, the E.S, found al the electical components to be in physically good eondition (ie. no burnt ‘or corroded wiring or terminals) He had identified during the inspection that the mechanical breaker ‘mechanism was in ned of lubrication, “The ES, identified and traced out the wiring from the outdoor F/G Tie breaker tothe back of Pane! 2, located inside the Sub 1 conteol building. From the back of Pane! 2, the E.S. identified the terminal block that the wires had landed on, and traced the corresponding wites tothe back of an auxiliary relay (SINX-G). The auxiliary relay SINX-G is activated by a corresponding electo-mechanical relay “Bus Section G Ground Overcurrent” GIN-G). ‘Arelay isan electical protective device, similar to a process safety relief valve. It is common for electro- ‘mechanical relays to be designed with visual “flags” indicating that a rely isin the tipped position. Unlike clecio-mechanical relays, auxiliary relays are commonly designed without a visual indication wien in the tripped position, and the state ofthe auxiliary relay would be checked by physically looking at the position of the relay contacis (Le. in the elosed or open position), ‘After tracing out the wires, the ES, returned tothe I/G Tie breaker in the early afternoon with three Eletricians to continue troubleshooting the breaker. An Electrical Joint Safety Analysis (JSA) was reviewed, completed and signed with the three electricians and ES. priar to commencing any additional troubleshooting work 2“The following sequence of events describes the troubleshooting that was done leading upto the loss of power to ‘Subs 1203 and 1206, and occurred inthe sequence outlined below: ‘¢ The 15K isolation switches for the F/G Tie breaker were physically opened to isolate the F/G Tie breaker from Bus Section G and Bus Section F (located on either side of the F/G Te breaker. ‘+ The FIG Tie breaker isolation switches were inspected once open and were visually in good condos ‘© The ES. and Plectriian #1 went inside Sub { control building to manually lose the F/G Tie breaker ‘control switch located on Panel 2 ‘+ Flectrician #2 and #3 stayed outside a the F/G Tie breaker to visually observe the operation ofthe breaker mechanism and observe if any components were bound up upon the control switch being activated ‘© AtI3:1445 the E.S. manually closed the FIG Tie breaker control switeh on Pane! 2 of Sub 1 and an ‘alarm annunciated on Panel 6 indicating “I2kV Bus Section Switch FIG Tie Trip” ‘+ Eleoticians #2 and #3 observed no mechanical movement inthe F/G Tie breaker, andthe breaker did not elose upon manually closing the F/G Tie breaker control switch. ‘+ Electrician #1 was told to silence the alarm by the F.S. at the alarm pane! which was located in Panel 6, approximately10 feet away from Panel 2, ‘© While Flectrician #1 responded to the alarm, the B.S. had thought that the alarm could be caused by a relay that was in the tripped positon. The E.S. thought, in particular, an auxiliary relay was possibly in the tipped position and removed the cover of auxiliary elay (SINX-G) on Pane! 2 ‘+ The ES. pulled out the auxiliary relay (51NNX-G) to check that the contacts were open and then pushed inthe auxiliary relay, withthe thought that this would change the state ofthe trip contacts, and the ‘alarm would clea. ‘+ Upon pushing in the auifiary relay SINX-G, the contacts ofthe SINX-G closed this immediately sctivated a tip of Bus Section G's main breaker (1700/2) resulting in loss of power to Sub 1203 and Sul 1206, These substations feed power to multiple units in LOP, OPCEN, Utilities and Logistics Vine Hill area ‘+ At13:15:39, “12kV Bus Tie TX-7 Switch 1700/2 trip” alarm came in on the slarm panel at Panel 6, indicating Bus Section G main breaker (1700/2) had tipped, + Hlectricians #2 and #3 heard noise ofa breaker opening outside of Subl + Approximately 30 seconds later, a call was received on the radio from Utilities operator that power was lost. + Once the E'S. confirmed the Bus Section G main breaker (1700/2) din fact tip, the E.S. performed. fanelectriel check on all associated equipment and components before proceeding to safely close the Bus Section G main breaker (1700/2). ‘© At3:17:57, Bus Section G main breaker (1700/2) was closed and power was restored o Subs 1203 fant 1706 Level 2 TOP/CL SMR2016.007 Although power was estored in about 2 minutes after the initial trip, many ofthe units had already begun ‘depressuring the units to the flare via te refinery safety systems.Level 2 TOP/CL SMR2016-007 5.0 Cause Analysis, ‘twas determined that Bus Section G main breaker (1700/2) tripped, resulting in loss of power to Subs 1203 ‘and 1206, due tothe manual closing of auxiliary relay’s (51NX-G) contacts, which sent a trip signal to Bus ‘Section G main breaker (1700/2) The investigation team focused the investigation on the human interfacing and decision making to determine the causes as to why the F.S. manually interacted with auxiliary relay (SINX- ©. ‘Te following eauses were identified as to Why the E.S. determined to manually close auxiliary relay SINX-O. ‘© The annuneiator “I2kV Bus Section switch FIG Tie trip” alarmed when the F/G Tie breaker remained ‘open, after manually netivating the control switch to the “elose” positon. ‘© The ES, believed there was a trip signal, likely ftom a relay, keeping the FIG Tie breaker from closing, ‘aasis the alatm description, ‘©The ES, believed that by mechanically closing the contacts of sul clear. ‘+ The E., had identified auxiliary relay SINX-G by tracing wires from F/G Tie breaker tothe back of Panel 2 in Sub 1 control building. The E'S. believed any action taken on Panel 2 was sae from consequence, yy relay SINX-G the alarm would ‘When the “12kV Bus Section switch F/G Tie tip” alarmed upon manually aetivating the control switch tothe “closed?” postion, the alarm description led the ES, to believe that tripped relay was possibly sending a signal to the FIG Tie breaker to remain open, Panel 2 has several electro-mechanical relays with corresponding auxiliary relays, The SINX-G auxiliary relay is activated by its corresponding electro-mechanieal relay, “Bus Section G Ground Overcurrent” (51N-G). The flag of electro-mechanical relay SIN-G remained in the un- ‘tipped position when the alarm snnuncited, which directed the E.S. to believe tht SIN-G’s auxiliary relay {GINX-G) could possibly be sending a “alse” trip signal, causing the alarm. The E.. believed that by ‘mechanically closing the contacts ofthe auxiliary relay, the alarm would clear. This would indicate that the ‘auxiliary relay (SINX-G) was sending a “false” trip signal tothe FIG Tie breaker which would cause it to remain open upon activating the control switeh tothe “closed” postion. “The auxiliary relay S1NX-G and tho electro-mechanical relay SIN-G are both located on the “12kV Bus Sectionalizing P&G” panel (Panel 2), The E.S. belived that manually closing auxiliary relay SINX-G was safe from consequence, because the E.S, believed the F/G Tie breaker components loeated on Panel 2 were isolated fiom the neighboring panels, Bus Section F, 1600/2 (Panet 1) and Bus Section G, 1700/2 (Pane! 3). The labeling ‘of the components on Panel 2 corresponded to the FIG Tie breaker and typically panels were organized by their ‘associated components. The F.S. also had reason to believe that manually closing auxiliary relay SINX-G on. Panel 2 wes aefe from consequence, becauoe the outdoor F/G Tie breaker was physically isolated from Ri Section G and F when the isolation switehes were manually opened in the field. See photo below of Panel 2 inside Sub |Level 2 TOPICL SMR2016-007 Bus Section FIG Tie Bus Section G (Panel 2) (Panel 3) Tigare ls Panes se Lastly, the B.S, had followed out the wires from his initial troubleshooting from the outdoor F/G Tie breaker into Sub | control building. These wires were chosen, because the ES. was avare of the alarm and FIG Tie breaker issue on December 13° and was looking for wiring that ran external from the F/G Tie breaker to Panel in Sub 1 control building. The wites were traced out and believed to land ona terminal block at the back of Panel 2, and were subsequently waced to auxiliary relay SINX-G. The wires that were identified and traced to ‘the back of Panel 2 were distinctive from the other wires, intemal and external t the F/G Tie breaker and Panel 2, in that they Were a pair of red and black wires, the wares had similar gauge, wire numbers, wie tags, and yellow wire lugs, See photo below of wies that were traced from the outdoor F/G Tie breaker tothe back of Panel 2Level 2 TOP/CL SMR2016-007 “Although isolated physically inthe feld, auxiliary relay SINX-G, located on Pane! 2, had active wiring to tip, Bas Section G's main breaker (1700/2) when manually pushed into th closed position. The trip contacts of auxiliary elay SINX-G to the F/G Tie breaker were shown on the associated I/G Tie breaker clectrical drawing, which the ES, had referenced during the F/G Tie breaker troubleshooting. he tp contacts of auxiliary relay SINX-G to Bus Section G's main breaker (1700/2) were included on a different craving. 66.0 Recommendations Level 2 TOPICL SMR2016-007 Recoramendation Responsible Manager oF Individual ‘Estimated Completion 1, Uplate Electrical Tob Safe Aaais GSA) template to include verbiage 10 Coal the ned for aitonal electrical Staff while troableshooting of break Controls and protective relaying in main substations Gabstation 3 90d 3) Electrical Engineer 8/31/2017 2, Install labels Tor proteetive relays on Panel in Sub I, Labels will identity the ‘corresponding breakers that will tip upon activation ofthe protective relay lectrial Engineer 22018 | 3. Install informational label Tor alarms associated with breaker trips on alarm panel (Panel 6) in Sub 1 Electrical Engineer 2sr2018 4, Review incident investigation learnings ‘with all applicable electrical personnel Causal Leaming Focal Paint 6802017