City officials know when water controls were tampered with, but not who did it

Cairo city officials have determined that last month someone purposely logged into the computer system that regulates the water level in a groundwater storage tank at the water plant on MacIvor Drive and then “acknowledged” a low level alarm, but took no action to correct the problem.
As it turned out, the city’s water system was drained nearly dry on Sunday morning, Jan. 8, 2012, because pumps used to pump water into the system were not automatically activated.
Although Cairo City Manager Chris Addleton knows now why the low water pressure event took place, he told city councilmen this week he cannot prove who logged into the computer system.
The city purchased the new Supervisory Control and Data Acquisition (SCADA) system in July 2010, and installed it in September 2010.
The city manager and members of the city staff and consulting engineer Stacy Watkins met with Tom Hopkins, CEO and vice president of sales for MR Systems, the vendor that provided the SCADA system. As a result of that meeting, the investigation into the incident revealed security issues and system weaknesses that have now been addressed.
According to Addleton, at 3:51 p.m. on Saturday, Jan. 7, someone logged into the system and changed the “well stop set point” for the groundwater storage tank from 21 feet to 0 feet.
Then at 2:51 a.m., the computer system shows that the “operator” acknowledged a “low level” alarm, but did nothing to activate the pumps at the MacIvor Drive water plant.
The reason the city manager cannot prove who was behind the incident is that four or five city personnel had access to the system using a common user name and password.
“I’m not going to make accusations that I can’t prove. Let’s move on,” the city manager told city councilmen Monday night.
Now, city personnel have unique user names and passwords so that a record of who logs in and when will be maintained.
“Now, we will have a ‘fingerprint’ on it, but most of the other changes we have made would make it impossible for anyone to do it again,” Addleton said.
In addition to the unique user names and passwords, the SCADA has been upgraded to communicate daily at 3:30 p.m. with the on-call staff member via voice mail and an email to the public works superintendent.
According to the city manager, this will indicate that the SCADA system has communication capability.
During the Jan. 8 incident, the SCADA system had lost communication since it was originally set up on a shared phone line that forced it to compete with a fax machine. The SCADA is now on a dedicated line.
A “dead phone line” alarm has been added and an “e-mail server connection” status alarm has been added.
Alarm settings for the groundwater storage tank increased to 10 feet for low and six feet for “low-low.” The settings at the Hilltop elevated tank have also been added and set for 80 feet for “low” level and 75 feet for “low-low.”
With the software upgrades, Addleton says the water level in the groundwater storage tank will never drop below 15 feet before the well pumps are activated.
As another precaution, a mechanical “low level” float switch has been added to the groundwater storage tank.
“These corrective actions will make the detection of and reaction to any low water pressure issue much more robust in the future,” Addleton said.
He added, “We could have caught it sooner than we did. I can’t guarantee it will never happen again, but the probability is a lot higher that we will catch it sooner if it does.”
Mayor Pro Tem James H. (Jimmy) Douglas added strong words to the city manager’s comments Monday night.
“If something had happened as a result of this, say a fire involving a fatality or total loss, the city would have been liable. There would have been a police investigation and the culprit would have been criminally prosecuted. Folks need to be aware of this. It doesn’t sound like much, but it is important. If we had had a fire and no water to put it out, there would be a lot of liability,” Douglas said.
“Everyone involved has had a talking to,” Addleton said.
Ironically, Cairo firemen were dispatched to a kitchen fire on the morning of Jan. 8, but the fire was quickly extinguished using a dry chemical rather than water.