DCS Console Reprogramming Causes Gateway Fault

Event Year: 1998 Reliability: Confirmed
Country: Canada
Industry Type: Pulp and Paper
Description:

The paper mill had just completed an upgrade of its paper machine, during which a number of engineers had been brought in from head office to assist with DCS commissioning. Everyone on the DCS commissioning team knew the passwords for the control system computers and when the project was completed, no one bothered to change them.

Trouble started about a month later when one of the head-office engineers decided he needed a good data source for an expert-systems experiment he was running. Using the company’s wide area network (WAN), he was able to connect into the mill network from the corporate headquarters several hundred miles away. Once into the mill’s business LAN, he was able to connect to the DCS through a link originally set-up to allow mill supervisors to view operators screens from their offices. He then loaded a small program onto one of the DCS graphics stations (a UNIX machine). This program asked all DCS devices to dump their data back to him once every five minutes.

Unfortunately the engineer’s new task would occasionally overload one of the DCS to PLC communications gateways, and the DCS would stop reading the PLC data. This, of course, caused the machine operators concern as they lost control of the motors controlled by the PLCs. Soon the electrical department was busy troubleshooting the PLCs.

Eventually the problem was solved by a mill engineer who noticed that the problems always occurred at intervals that were at multiple of five minutes. Suspecting that it might be software induced, he started to inspect all the tasks running on the DCS computers and found the offending task. Of course, by then the lost production in the mill had been substantial.

Action Description: Installed VLAN system and Firewalls between business and process networks