Ethernet Network Storm Zaps Multiple PLC5’s

Event Year: 2003 Reliability: Confirmed
Country: United States
Industry Type: Pharmaceutical
Description:

The company has an Ethernet network of 42 PLC5-E (5/20, 5/40) with redundant SCADA servers. PLCs are grouped, 3-5 together on coax 10base2 trunk/drop lines. These then go thru media converters to redundant Cisco switches with redundant fiber uplinks. The SCADA network and operator terminals are also on network, but not connected to the IS network or the outside world.

An intermittently faulty fiber caused the switches to generate multiple Spanning Tree Protocol discovery messages. The number of these escalated over several hours until one switch locked up and nothing would move on the network. (the switch terminal port locked up as well). At the same time, 13 PLC’s on several different trunk lines faulted with solid red fault light. All PLC programs were wiped and the PLCs required a power cycle before even serial comms were possible.

Rockwell PLC5 Release Notes (1) state that “Series C, revision H and later processors limit the amount of messages they will accept under extremely high levels of Ethernet traffic (storms). This is designed to prevent a fault with memory loss.” The PLCs in question were Series C Revision E. Thus it is likely that this incident was due to the Ethernet traffic storm caused by the STP flapping.

Impact:

Lost production for 2 _ hours and a batch in process had to be destroyed.

Action Description: All PLCs were upgraded to a newer version of firmware.