At power- and water-generating facilities, it’s important that operators do not miss a beat. So many citizens depend on having access to electricity, heat and clean water that in the event such service is disrupted, the outcome is highly undesirable.
Publications Security and SDM brought together five security professionals — three practitioners and two systems integrators — who are involved with security in the Utility & Critical Infrastructure sector, to discuss the unique needs they have and the solutions that work for them. Some areas of discussion: outdated technology and how to upgrade it; an increasing level of regulation and the need to comply; threats of terrorism, natural disaster and routine outages — and the importance of business continuity.
In the following discussion, conducted by Security editor, Diane Ritchey, and SDM editor, Laura Stepanek, security professionals offer an insight into this specialized world.
Diane Ritchey: Let’s begin by asking about what is happening in the area of critical infrastructure protection today, and what is your role?
Thomas Collins: For most large water utilities, the security systems are in place and we are now in the second or third phase of expanding or improving the current legacy security system. The goal is to maintain the legacy system and migrate to a hybrid analog/IP driven system, eventually moving to a complete IP system. It should not be forgotten that the current security systems were put into place based on recommendations in the Vulnerability Assessments of 2003. Security technology has exponentially grown since 2003 so essentially utilities have to improve and expand while maintaining security.
Moving to an IP network security system is essential because it will allow us to cover extended areas with CCTV, improve efficiency in maintenance scheduling, and generate reports, summaries of video views, instant reporting of downed cameras, all of it a part of the network system integrated with IT and the corporate structure.
The new systems allow for coverage with intelligent video analytics that can replace personnel who will no longer be required to watch many monitors, in essence fulfilling the motto of Allan Pinkerton’s Agency, “We never sleep.”
For large cities the
ultimate goal is secure WiFi and the City of
Laura Stepanek: How many analog cameras do you have at the water utility?
Thomas Collins: We’re looking at several thousand.
Laura Stepanek: Have you estimated the cost for such a replacement?
Thomas Collins: Yes. It’s pretty staggering. But at the time that we were putting our system together, the Bioterrorism and Response Act mandated that we have a vulnerability assessment in place and report it to the EPA by September 2003. So we signed off on what we were able to put together at the time using existing technologies, and at the time I think analog was pretty much the standard, at least in the water utility area.
Karl Perman: Many electric utilities are working on implementing NERC, the North American Electric Reliability Corporation, Critical Infrastructure Protection Standards. These standards basically cover the fundamental security requirements for both cyber as well as physical security. The standards were created to better protect the bulk electric system.
Diane Ritchey: So it’s increasingly become a more regulated environment?
Karl Perman: Yes. And there are hefty fines if you don’t comply. NEARC has put out one fine schedule that can lead to as much as $1 million dollars per day per instance of non-compliance.
Diane Ritchey: And what are the biggest challenges in meeting those compliance standards? Is it money? Is it time? Is it both?
Karl Perman: Without going into specifics, in general, it is resources, because you are going from areas that were not regulated in the past to being regulated. So as you might imagine, it’s a new world.
Laura Stepanek: David, tell us about your challenges working with the Prairie Island Nuclear Generating Camp.
David Axt: Yes, it is. By the way, Karl, I feel your pain, so-to-speak. Because since 1979 there have been incremental increases in security requirements. So virtually everything that we do is regulated. The federal regulations specify that you have to be able to defend against this or that threat, and then it is up to us to find the particular upgrades.
We just finished a new code of federal regulation changes. While I can’t get into the specifics, I can say that we’re essentially putting in place physical security upgrades as a result of new terrorist scenarios.
Some of the other big changes for us include the number of hours that armed nuclear security officers can work at any given time. And cyber security is another change. It’s not true what you see in the movies, by the way — you cannot shut down the plant using the Internet! Cyber security for us essentially involves applying physical security to certain cyber resources here on site.
Laura Stepanek: Let’s hear from the integrators on this topic. I think they are going to obviously have a different perspective. Ed, what is your company’s role in this area of critical infrastructure protection?
Edward Newman: We’re a security systems integrator in the New York Metro area. Over the last couple of years about 50 to 60 percent of our revenue has come from the utility market. We work with a number of the larger utilities in the area, and we have been doing for almost 20 years now.
After 9/11 we saw a rush by many utilities to put money into security. As Tom and Karl have brought up, regulation is really starting to change the industry.
We’re seeing a lot more of the projects driven by regulatory requirements. And we are frustrated with seeing much of this based on threat assessments and facility security plans that were written immediately post 9/11. We’re often now being asked to work on projects that are utilizing technology that was spec’d in 2001 or 2002. In some cases it was actually late ‘90s technology. But there’s better technology available. Yet we seem to be at a stagnant point; we are not moving forward.
We have plenty of customers that are still installing analog solutions, even though in some cases they know that the IP solutions are what they need to do instead.
Laura Stepanek: And why is that happening, Ed? Why are those projects just now being intsalled when they were spec’d so long ago?
Ed Newman: It’s the unavailability of funds. It takes a lot of money to secure a utility, whether it is a generation plant or the various transmission facilities. In many cases, it might cost $100 million to secure a utility. Most of the customers that we deal with are under budget pressure right now. How can they reduce their operating costs? And many of them haven’t come to the realization that while they may have spent $50 million since 9/11 installing security, there is a cost obligation that goes with maintaining and upgrading those systems.
Dave Axt: I have heard of some facilities putting in multi-million dollar physical security upgrades, intrusion detection systems, or early warning devices and not fully considering the operating maintenance aspects or the repairs or extra personnel that will maintain the systems.
Thomas Collins: One thing I know that is being considered is moving costs for maintenance of systems to other groups within the utility. So if security gives up part of their control of their security systems over to the IT department, then IT absorbs the cost as well.
Laura Stepanek: Misty, can you share with us what’s happening in your organization?
Misty Stine: We are a national systems integrator. We do design into play systems for water-wastewater facilities as well as electrical grid facilities.
I agree with all of the things that we have discussed with respect to integrating legacy systems with new systems, in addition to how to maintain your capital investment that you have already made in past security systems.
What is important to keep in mind is that all of these systems are software based. You have to do regular software upgrades in order to keep a system running optimally.
One thing that we are looking at now is pending legislation that will impact the water-wastewater industry. We’re also working closely with our electrical grid customers in continuing to place security measures that protect their control centers.
Our role as a systems integrator is not only to design and deploy systems, but also to help our customers seek out new technologies like command-and-control platforms, or dashboards. There are a lot of systems out there, including one that can actually overlay your legacy systems and allow one common platform that your users can view at any given time, but yet also integrate all of your systems together regardless of the type of system. There are ways to protect your investment.
Laura Stepanek: Our national infrastructure is vulnerable to a combination of threats, both traditional and non-traditional. These could include natural disasters, physical attacks, cyber attacks, human error, and the list goes on and on. Let’s ask each of you to elaborate on each of these threats in a general way and explain what the current thinking is about how to best protect against each one.
Thomas
Collins: Actually, our biggest threat is hurricanes. We deal with them
in
Another concern is a connection between how we look at our emergency response plan regarding a natural disaster as opposed to say physical attacks. There is going to be some kind of relation. It’s just going to be on a different scale and with a different emphasis.
Karl Perman: We take an all-hazards approach. The approach includes a combination of protections, first which is having plans in place, such as the appropriate disaster recovery plans and business continuity plans for natural disasters as well as other hazards such as physical or cyber attacks. As far as systems, we use a combination of technologies, from closed circuit television systems to intrusion detection systems.
Diane Ritchey: What about you, David? What is your experience in this area?
David Axt: Nuclear facilities are required to be designed to protect against natural disasters. The inherent facility design can protect against flooding, tornadoes and earthquakes. And, of course, nuclear facilities are also required to have emergency plans for such events.
Cyber attacks are not so much a problem for us here at the nuclear facility, I should note. Our primary threat is a terrorist threat, an armed attack that is dedicated to highly motivated military trained force bent on sabotaging a target.
But I think you find that at a nuclear facility we have the classic physical security elements: we have deterrents, signage, fences placed further out from our protected area and such. We have detection and assessment, delay barriers and a response plan in place.
But what I see working well for a nuclear facility is adding early warning intrusion detection systems even further out from a standard double-fenced protected area. It’s a multiple layered system. Most nuclear facilities are required to use some sort of volumetric type of intrusion detection to protect the perimeter.
Misty Stine: There are a number of things that David and Karl have described as it relates to natural disasters and physical attacks and the layered approach. I think that a layered approach is not only manned guards as was described, but is one that starts with a manned guard solution and then includes perimeter security and access control. All of those elements are important.
With relation to a
natural disaster, it is also important that you make sure that you select a
solution that meets your unique environment. What might work for you in
And proper maintenance of the system is also important. Karl mentioned that he uses a business continuity plan, but I think that’s something that often is overlooked by our users. It’s important that you have an emergency response plan, but what do you do once the emergency is used? How do you get your business back online?
Ed Newman: I agree. We have seen a trend in the last year when it comes to determining what the threats are and how they need to be protected against. There is more analysis of threats and the implications of an attack. And while a successful attack on a nuclear facility could easily be considered a mass casualty incident, the majority of the attacks that could happen to electric or gas or water utilities are generally more quality-of-life interruptions.
Yet at the same time, if your water or sewer facility is down for longer than a few hours, or even days, that’s a problem. If it is down for a week, the implications can be tremendous. And you can very quickly get to a loss of life and complete chaos.
Thomas Collins: Hurricane Ike shut down our systems. We were without wastewater for 44 days. So nine days after Hurricane Ike, there were 250,000 people that did not have running water. And within 75 miles up the coast there were 1,300 community water systems that were down. So there were close to 1,000 or 1,200 boil-water notices issued after Hurricane Ike for the Coastal Region. It was devastating for us. And it was a struggle for us to literally rebuild these systems from the ground up while we were still trying to provide water and wastewater treatment for the citizens.
Ed Newman: I liked to hear that utilities are focusing more on business continuity. We are seeing that as well, as the majority of our clients have a significant concern with business continuity and what happens in a variety of scenarios.
The one thing that we have not seen is that it doesn’t seem to go beyond the immediate first layer, for example, when you rely on contractors to fix your systems in an emergency situation. How many of those contractors were actually available at that point, due to damage to their own facilities? Did they have a plan in place to respond to you?
Thomas Collins: We have emergency contracts for our vendors, but in an emergency situation it is about your relationship with your vendor, and whether or not that vendor is out of state or locally, and if they are going through the same disaster as you. So with our continuity of operations plans, sometimes it’s best for us to look for vendors that are out of state.
Misty Stine: I think that’s a great statement in that you need to look for redundant sources: a redundant network, command center and remote monitoring capabilities so that if you do have such an emergency as you did in Houston that you have the ability to move your traffic or sensors or alarms somewhere else to be monitored or to be addressed in another location.
Laura Stepanek: What are the best ways to have public and private concerns working together for the best interests of your particular facility or your organization?
Karl Perman: That involves a combination of things. First, is
working with the sector coordinating councils that are part of the government
coordinating councils in conjunction with the Department of Homeland Security.
Second, it means getting involved with industry associations. We also have
great relationships with the FBI as individual state, county and local public
safety
agencies.
Thomas Collins: The best policy that works for us is one that is part of the overall planning process for the utility. For instance, our ERP is directly related to our vulnerability assessment. The vulnerability assessment was conducted using all stakeholders that had an interest in keeping the water/wastewater utility system operating and secure. That is the time for complete buy-in to the security policies, when there is complete engagement by the entire executive and upper management levels into the process.
Continuity of the vision, however, changes when the executive changes, so the vision is always undergoing “re-vision.”
Ed Newman: Because we are involved with both the design of the system, the installation, the support and the operation of the system, while utilities and critical infrastructure business are developing good strong partnerships with the local emergency services, emergency management agency, we often don’t have a seat at the table. We’re seen more as just a commodity provider. And it’s challenging because often there is a lack of the detailed technical understanding of what goes into these systems in those partnerships. And that’s the piece that we would bring.
So the relationships that we see that work best are when we as the integrator are a partner with the utility, and we are involved in all levels of the security management and planning process.
Misty Stine: I think it’s also important that end users share their concept of operations with the integrators. It’s very important for us to understand the concept of operations in order to apply and deploy the appropriate solution. So often a missing piece of the puzzle is having a well-defined concept of operations, and then sharing the concept of operations with your integrator and partner.
Laura Stepanek: Our last question relates to your wish list. What is on your wish list for this sector for the next year or so?
Thomas Collins: I wish for more cooperation and communication between utilities. The utilities currently share information from WaterISAC, and the information provided by WaterISAC is timely, and intelligence driven; however, I see a shortfall in shared intelligence and resource information — something that I imagine others in my position are reluctant to discuss since many security managers are not willing to openly discuss security concerns, innovations and plans even among their peers. This is something I would like to see change and am currently working with WaterISAC on one of their advisory boards.
Karl Perman: On my wish list are more resources.
Ed Newman: I will second Karl and add that funding is always a problem. We have customers that are probably receiving 90 percent of the money that they are spending on physical security through grants. And we have ones that are getting zero grants.
The grant process is complicated. We have customers that have applied numerous times to be rejected, not on merit, but on improperly completed paperwork. The competition for qualified individuals to write those grants is extremely high. So it’s hard to get somebody to do that.
Money aside, I probably would say the biggest piece where we have seen it take a step backwards, and I would like it turned around is more involving the integrator in the process, and looking at them as a partner in providing the solution than as a commodity. As we separate it out into a commodity we have seen more where we are asked to provide systems that we know have flaws in them or we know have problems.
David Axt
David Axt has 25 years of experience in nuclear security operations, to include supervision, management, auditing, contingency planning, risk analyses and training. He is currently the security manager at Xcel Energy’s Prairie Island Nuclear Generating Plant. He has a Master of Science (MSc) in Security & Risk from the University of Leicester, England. He is certified in Homeland Security – Level III, American College of Forensic Examiners Institute. He is also an adjunct professor with Concordia-St. Paul’s Criminal Justice Department.
Tom Collins
Thomas Collins Ph.D, is security manager for the City of Houston’s Public Works & Engineering Dept., responsible for managing the security for the fourth largest Water/Wastewater Utility in the country. He has two Masters Degrees and a Ph.D. in Urban and Regional Science from Texas A&M University. He has taught and facilitated for four years with the University of Phoenix systems.
Edward J. Newman
Edward J. Newman is vice president with Universal Security Systems, Inc. He has a Bachelor of Science degree from Polytechnic University. He is a licensed Master Electrician and he holds various trade licenses in three U.S. states.
Misty Stine
Misty Stine is the vice president of business development for Adesta. She has senior management responsibility for the company’s strategic corporate initiatives and vertical market campaigns. Misty joined Adesta in 1995; in 2005 she played a pivotal role in the creation of the Adesta Critical Infrastructure division. Misty has more than 25 years of experience in both the communications and security industries. She attended Arizona State University where she received her Bachelor’s degree in Social Sciences.
