HomeMy WebLinkAboutData Processing Disaster Recovery PlanRESOLUTION 2011-366
ADOPTED
DOC ID: 6852
THIS IS TO CERTIFY THAT THE FOLLOWING RESOLUTION NO. 2011-366 WAS
ADOPTED AT THE REGULAR MEETING OF THE SOUTHOLD TOWN BOARD ON
MAY 10, 2011:
RESOLVED that the Town Board of the Town of Southold hereby adopts the Town of
Sonthold Data Processin~ Disaster Recovery Plan Policy, effective immediately.
Elizabeth A. Neville
Southold Town Clerk
RESULT: ADOPTED [UNANIMOUS]
MOVER: William Ruland, Councilman
SECONDER: Vincent Orlando, Councilman
AYES: Ruland, Orlando, Talbot, Krupski Jr., Evans
ABSENT: Scott Russell
TOWN OF SOUTHOLD
DATA PROCESSING
DISASTER RECOVERY PLAN
REVISION 1.0 April 26, 2011
Introduction
Data Processing (DP) services are now critical to the business mission of the Town of Southold. As a
result of this ever increasing reliance on technology, DP services require a comprehensive Disaster
Recovery Plan to ensure these services can be re-established quickly and completely in the event of a
disaster. This plan provides general steps that should be considered in the event of a disaster to
restore DP functions. It touches on provisions and recommendations for restoring the technology
infrastructure that require executive level management approval and additional funding to implement.
Objectives
The primary objective of this Disaster Recovery Plan is to help ensure business and technology related
continuity by providing the ability to successfully recover computer services in the event of a disaster.
Specific goals of this plan relative to an emergency include:
Detailing a general course of action to follow in the event of a disaster,
Minimizing confusion, errors, and expense to the Town.
Implementing a quick and complete recovery of services.
Reducing risks of loss of services,
Providing ongoing data backup services.
Scope
This plan addresses the recovery of systems under the direct control of the Data Processing
Department that are considered critical for business continuity. Also, given the uncertain impact of a
given incident or disaster, it is not the intent of this document to provide specific recovery instruction
for every system. Rather, this document will outline a general recovery process which will lead to
development of specific responses to any given incident or disaster.
Southold Town Data Processing Equipment Locations
In the event of a disaster, the Town data processing equipment is located in the following areas.
Town Hall Server Room, Hardware, Application and Software
Town Hall Phone Room and Phone Closet
Police Department Server Room, Hardware, Applications and Software
Police Dispatch Room
Town Hall Annex Computer Closet, Hardware, Applications and Software
Network Infrastructure and Services
In addition to the areas mentioned above the following equipment may be affected in the event of
disaster:
Personal Computer Hardware, Applications and Software
Data Backup
The potential risks to each area are discussed in further detail below.
Assumptions
This disaster recovery plan is based on the following assumptions:
2 [ I:::~ ;ii~/iil; e
The safety of human life is paramount; the safeguard of such will supersede concerns specific
to hardware, software, and the recovery needs.
Once an incident covered by this plan has been declared a disaster, the appropriate priority will
be given to the recovery effort, and the resources and support required as outlined in this
disaster recovery plan will be made available.
Depending on the severity of the disaster, other Town departments may be required to modify
their operations to accommodate changes in system performance, computer availability, and
physical location until a full recovery has been completed. The Town Board should encourage
departments and committees to have contingency or business continuity plans for their
operations, which include operating without DP systems for an extended period of time.
The Network and System Administrator staff will be responsible for: Overseeing damage assessment
Development of a recovery plan specific to the incident
Schedule and set priorities required to mitigate the disaster.
Restoration of the affected area to pre disaster functionality including the space,
hardware, applications and software.
Risk Assessment
Town Hall Server Room
The Town Hall building is a one story, wood frame structure located at 53095 Main Road, Southold NY
'1'197'1. The Data Processing staff, in its entirety, is housed in the east side of the facility on the '1st
floor. The server room is located in the basement of the structure with only one means of access and
egress. This room houses the Town's main servers and network equipment. Except for the Police
Department it is the location where all data and transmitted communications for Town of Southold
Data Processing are redirected, combined, stored, and retrieved. Data backups are stored daily to disk,
weekly and monthly to tape. The tapes are kept off site in a secure cabinet located in the Town Hall
Annex building. Disk storage units are located in the server room. There is no offsite backup facility
currently identified, that could replace the functions of the Server Room if it is rendered in operable by
environmental or an accidental disaster.
Risks/Declarations
· There are numerous employees and officials that have a master key which can unlock the
exterior of the computer room and a few that, have a key for the inner door.
· There is an environmental sensing device installed in the computer room to detect water.
· The building structure is wood frame; it houses computer equipment, has numerous storage
areas and has office space and individual cubicles which contain documents, books and
equipment.
· The server room contains large quantities of equipment, but minimal combustibles such as
papers or documents.
· The building does have a wet stand pipe system for fire extinguishment.
· An internally mounted A/C unit provides cooling throughout the year.
· The A/C unit does not have heaters; however the computer equipment in the phone room
produces heat, so the risk of too Iow a temperature is minimal.
· There is a temperature sensing device that alerts the Data Processing staff of spikes in
temperature.
· Power is provided to the building from LIPA through the regular power grid.
· Standby power is currently provided by a diesel generator for the building.
· UPS systems are in place to maintain power until the generator is activated and for safe
shutdown purposes only.
Recovery Planning
· Recovery decisions will be based on the extent of the damage to the Town Hall and server
room. A Hot backup computing facility does not currently exist, so if the server room remains
habitable, every effort will be made to re-establish service in the same area.
· If the Town Hall is not habitable then every effort will be made to set up a temporary server
room in one of the other Town locations.
· If the server room is not habitable then every effort will be made to set up a temporary server
room in one of the other offices in Town Hall.
· If it appears recovery of individual services will take longer than a week to restore, on a
selective basis, services will be evaluated for possible outsourcing to commercial
organizations.
Town Hall Phone Room and Phone Closet
The Town Hall building is a one story, wood frame structure, located at 53095 Main Road, Southold NY
'1'197'1. The Data Processing staff, in its entirety, is housed in the east side of the facility on the '1st
floor. The phone room and phone closet are located in the basement of the structure. The phone room
has only one means of access and egress, while the phone closet has 2 means of access and egress.
These areas house the Town Hall's phone communication equipment. There is no other facility that
could replace the functions of the phone room or phone closet if they are rendered in operable by
environmental or an accidental disaster.
Risks/Declarations
· Both rooms are left unlocked and there are no windows in either area.
· The building structure is wood frame; it houses computer equipment, has numerous storage
areas and has office space and individual cubicles which contain documents, books and
equipment.
· The areas contain large quantities of phone equipment and cabling. Small, contained fires
are possible in the wiring and equipment.
· Storage of combustibles (cardboard, paper, plastics, liquids) is not allowed in the server
room.
· The building does have a wet stand pipe system for fire extinguishment.
· Power is provided to the building from LIPA through the regular power grid.
· Standby power is currently provided by a diesel generator for the building.
· UPS systems are in place to maintain power until the generator is activated and for safe
shutdown purposes only.
Recovery Planning
· Recovery decisions will be based on the extent of the damage to the Town Hall, phone room
and phone closet. A Hot backup computing facility does not currently exist, so if these areas
remain habitable, every effort will be made to re-establish service in the same area.
· If the Town Hall is not habitable then every effort will be made to set up a temporary
communications in one of the other Town locations.
· If the phone closet is not habitable then every effort will be made to set up a temporary
communications in one of the other offices in Town Hall.
· If it appears recovery of individual services will take longer than a week to restore, on a
selective basis, services will be evaluated for possible outsourcing to commercial
organizations.
Police Server Room
The Police Department building is a one story, concrete block structure, located at 4'1405 RT 25
Peconic, NY '1'1958. The Data Processing staff maintains a workstation in the server room which is
manned '1 shift per week. The server room is located in the basement of the structure with only one
means of access and egress. This room houses the Police Department's main servers, network and
communication equipment. This location is where all data and transmitted communications for Town
of Southold Police Department are redirected, combined, stored, and retrieved. There is a redundant
server located in the Town Hall server room that is used only for computer aided dispatch redundancy.
Aside from that, there is no offsite backup facility currently identified that could replace the functions
of the server room if it is rendered inoperable by environmental or an accidental disaster.
Risks/Declarations
· The server room door is left unlocked since the building itself is a secure facility. There are
no windows in the server room.
· The building houses computer equipment has numerous storage areas and has office space
and individual cubicles which contain documents, books and equipment.
· The server room contains large quantities of equipment, but minimal combustibles such as
papers or documents.
· Storage of combustibles (cardboard, paper, plastics, liquids) is not allowed in the computer
room.
· An internally mounted A/C unit provides cooling throughout the year.
· The A/C unit does not have heaters; however the computer equipment in the server room
produces heat, so the risk of too Iow a temperature is minimal.
· There is a temperature sensing device that alerts the Data Processing staff of spikes in
temperature.
· Power is provided to the building from LIPA through the regular power grid.
· Standby power is currently provided by a generator for the building.
· UPS systems are in place to maintain power until the generator is activated and for safe
shutdown purposes only.
Recovery Planning
· Recovery decisions will be based on the extent of the damage to the Police Department and
server room. A Hot backup computing facility does not currently exist, so if the dispatch room
remains habitable, every effort will be made to re-establish service in the same area.
· If the Police Department is not habitable then every effort will be made to set up a temporary
server room in one of the other Town locations.
· If the server room is not habitable then every effort will be made to set up a temporary server
room in one of the other offices in the Police Department.
· If it appears recovery of individual services will take longer than a week to restore, on a
selective basis, services will be evaluated for possible outsourcing to commercial
organizations.
Police Dispatch Room
The Police Department building is a one story, concrete block structure, located at 4'1405 RT 25
Peconic, NY '1'1958. The dispatch room is located on the southwest side of the structure on the first
floor with two means of access and egress. This room houses the Police Department's dispatch
computers, radios transmitters and related equipment. Aside from that there is no offsite backup
facility currently identified that could replace the functions of the dispatch room if it is rendered
inoperable by environmental or an accidental disaster. However, mobile units have the ability to
communicate with the redundant computer aided dispatch application server located at Town Hall and
are able to dispatch as well.
Physical/Security Risks - The dispatch room doors are left unlocked since the building itself is a
secure facility and the room is occupied 24 hours a day every day. There are 2 windows in the dispatch
room, one on the west wall and one on the south wall of the room that is made of bullet proof glass
and is used to communicate with people in the lobby. The outer windows are susceptible to breakage
and possible unauthorized entry.
Risks/Declarations
· The dispatch room contains large quantities of equipment, has combustibles such as papers
or documents.
· The room contains large quantities of equipment, but minimal combustibles such as papers
or documents.
· An internally mounted A/C unit provides cooling throughout the year.
· Power is provided to the building from LIPA through the regular power grid.
· Standby power is currently provided by a generator for the building.
· UPS systems are in place to maintain power until the generator is activated and for safe
shutdown purposes only.
Recovery Planning
· Recovery decisions will be based on the extent of the damage to the Police Department and
server room. A Hot backup computing facility does not currently exist, so if the dispatch room
remains habitable, every effort will be made to re-establish service in the same area.
· If the Police Department is not habitable then every effort will be made to set up a temporary
dispatch room in one of the other Town locations.
· If the server room is not habitable then every effort will be made to set up a temporary dispatch
room in one of the other offices in the Police Department.
· If it appears recovery of individual services will take longer than a week to restore, on a
selective basis, services will be evaluated for possible outsourcing to commercial
organizations.
Town Hall Annex Computer Closet
The Town Hall Annex building is a two story, masonry structure located at 54375 Main Rd, Southold,
NY 11971, in the rear of Capital One Bank building. Although the Data Processing staff makes frequent
trips to the location it does not maintain an office there. The server closet is located on the southwest
side of the structure on the second floor with two means of access and egress. This room houses the
Annex's communication, and network equipment. All server and data is stored off site in the Town Hall
server room.
Physical/Security Risks - There are numerous employees and officials that have a master key, which
can unlock the computer closet. There are no windows in the computer closet.
Risks/Declarations
The building structure is a block and masonry structure, it houses computer equipment, has
numerous storage areas and has office space and individual cubicles which contain
documents, books and equipment.
6 I I:::~ ~ Il; ~
· The computer closet contains large quantities of equipment, but no combustibles such as
papers or documents. Widespread fire is not likely in the computer closet; however, small,
contained fires are possible in the wiring and equipment.
· Storage of combustibles (cardboard, paper, plastics, liquids) is not allowed in the computer
closet.
· There is no A/C unit to provide cooling in the computer closet. The room is small and
contained which tends to lend itself to becoming hot.
· Power is provided to the building from LIPA through the regular power grid.
· UPS systems are in place to maintain power; however they will not last long since there is
no generator backup.
Recovery Planning
· Recovery decisions will be based on the extent of the damage to the Town Hall Annex and
computer room. A Hot backup computing facility does not currently exist, so if the computer
room remains habitable, every effort will be made to re-establish service in the same area.
· If the computer room is not habitable then every effort will be made to set up a temporary
computer room in one of the other offices in the Town Hall Annex.
· If it appears recovery of individual services will take longer than a week to restore, on a
selective basis, services will be evaluated for possible outsourcing to commercial
organizations.
Network Infrastructure and Services
The Town's network services are provided by a wired and wireless network infrastructure. Network
services include a wide variety of functions, such as network/file storage (including the associated
backup), printing, routing, switching, DNS, and DHCP services, web/internet services, bandwidth
allocation and monitoring, firewalls, etc. These services are totally dependent on a wide variety Town
owed or other commercial equipment including servers, switches, routers and wireless access points.
Each of the Town facilities contains some type of network servicing equipment with most of the
supporting equipment located in the Town Hall server room.
Risk Assessment
Physical/Security Risk
· With the exception of the switching electronics and equipment located in the various town
buildings, all equipment supporting network services is located in the Town Hall Sever Room.
· Currently there is an offsite data storage capability at the Town Hall Annex located on Route 25
in Southold. Once a week the data is backed up to tape and stored offsite. Data located on any
disk backup system would be lost if the server room was rendered inoperable.
· Telephone and data switching electronics are located in the Town Hall server room and phone
closet and/or wiring closets located in each of the Town's buildings.
· These areas are also used for miscellaneous storage and are accessed by other than ITS
personnel.
· The risk for inadvertent damage and possible malicious damage is medium in these areas.
· Many of these areas are in environments that tend to be excessively dusty/dirty and suffer from
significant humidity and temperature fluctuations. This can cause a higher than normal network
electronic failure rate and reduce the lifetime of the copper network and telephone
termination/cabling.
Environmental Risk
· Wiring closets and rooms are generally not environmentally controlled and subject the
equipment to varying humidity and temperature extremes and exposure to excessive dirt and
dust. There is a risk of equipment and cabling failure because of the lack of a reasonable
operating environment.
Internal Systems Risk · Hardware or software failure impacting individuals at remote location network services is a
significant risk and would require significant cost to replace or improve.
· Most network services do not have redundant hardware or failover systems in place. There are
numerous unique hardware items that represent potential single points of failure.
· Some of the equipment is used beyond its support life.
· UPS systems are in place but are not regularly tested and/or replaced.
External System Risk
· Internet connectivity is dependent upon Lightpath fiber connections; one to the Town Hall, one
to the Town Hall Annex and one to the Police Department. Internet connectivity for all other
Town locations is depended upon Cablevision connections. All of these connections can be
damaged, resulting in the loss of external connectivity.
· There are currently no secondary (backup) data lines between the buildings.
· Hackers could attempt to launch denial of service attacks and/or attacks against network
equipment and/or configuration files.
Recovery Planning
· Given the wide variety of potential problems that could impact network services, the following
generic recovery planning steps will be utilized to identify and resolve network problems:
Assess which network service or services have been lost.
Notify Town employees, by whatever means available, as to the service outage.
Troubleshoot to isolate the cause of the service outage - if necessary, contact the
appropriate vendor for diagnostic support.
Once the problem is isolated, take appropriate action to restore the service(s).
In the event the service cannot be restored in a timely fashion, assess possible
workarounds, including temporary outsourcing, if feasible.
~ Notify the employees as to the status of the affected service.
o Notify the employees when the service becomes available.
Preventative Measures · Maintenance agreements are maintained on all critical servers and systems to help mitigate the
lack of redundancy and to ensure rapid vendor response to problems.
· Ensure that annual vendor maintenance agreements are in place for all critical network
systems.
· Funds permitting, replace obsolete equipment and improve backup hardware.
· Obtain funding for redundant Town wide connections to all building wiring rooms and closets.
· Maintain offsite storage/backup of configuration files.
· Funds permitting build up and maintain a stock of wiring closet and room hardware
· Improve and standardize backup power to switches located in wiring closets and rooms.
· Where possible, do not use wiring closets for storage purposes.
· If possible, buildings, wiring closets and rooms should be secured and locked at all times.
· Improve climate in wiring closets where there are significant temperature fluctuations.
· Adequate training and career growth opportunities must be provided to maintain the Town's
current technical staff.
· Funds permitting, offsite disk storage capability should be developed for all servers.
· Given the user requirement for 24/7 network and internet availability, establishing high
availability redundancy for all connectivity services should be budgeted for and implemented to
minimize loss of services.
File Sharing Storage and Services
The Town uses Microsoft Windows, IBM OS400 and Linux based servers to provide file sharing,
storage and related services. These servers provide users networked disk space to store files in
personal home directories and collaborative group directories. Documents, spreadsheets, databases,
and other digital information and programs store and retrieve data from these servers.
Risk Assessment
Physical/Security Risk
Servers are physically located at the Town Hall, Police Department and at the Annex. One Window
server is located at the Annex and multiple servers are located in the Town Hall and the Police
Department server rooms
Internal Risk Assessment
· In the event application software is lost due to equipment malfunctions, all required application
and operating system software could be obtained from the vendor.
· All current production servers are covered under a manufactures hardware warranty or with an
outsourced hardware warranty vendor.
· The most significant software-related risk is associated with losing data stored (not the system
state) on these file servers. To mitigate this risk, the following backup approach is currently in
place which accommodates the disaster recovery needs:
Monthly server full data to tape backups are stored offsite in a fire resistant cabinet for
one year.
Weekly server full data to tape backups are stored offsite in a fire resistant cabinet and
are rotated every other Friday.
Weekly server full data to disk backups are done every Saturday.
Daily server full data to disk backups are done Monday through Thursday to storage
devices that are physically located in the same room as the servers.
Most but not all of the personal computers are backed up to storage devices that are
physically located in the same room as the servers.
External Risk Assessment
· Network connectivity is vital to the functionality of all of the Town's networked devices. The
Town cannot operate without a functioning network.
Recovery Planning Assessment in which server service or services have been lost. · Notify the user base of the service outage.
· Troubleshoot and isolate the cause of the service outage - if necessary, contact the appropriate
Vendor(s) for diagnostic support. Once the problem is isolated, take appropriate action to
restore the service(s).
· In the event the service cannot be restored in a timely fashion, assess possible workarounds.
· Notify the user base as to the status of the affected service.
· Notify the user base when the service becomes available.
Preventative Measures
· Current preventative measures include:
o Maintenance contracts are maintained on all servers and networking hardware during the
operational life of the equipment.
o Hardware and software patches and upgrades are installed on a regular basis.
~ Backups are performed on a regular basis.
· Future preventative measures include:
Investigate the possibility of server clustering/high availability options to provide automatic
failover and system redundancy in the event of hardware failure.
~Investigate network redundancy.
~Develop a test environment to periodically restore and test backup's operational status.
Email services
Email service includes email delivery, virus scanning, spam blocking and email storage. Currently,
only Elected Officials, Appointed Officials, Committee Members and Employees are allowed to obtain
an email account.
Risk Assessment
· There is always the possibility of a physical/security risk of an attacker gaining physical access
to the email servers; in this case all security measures could be bypassed.
· The Town email servers are located in the Town Hall Server Room.
· See the Town Hall Server Room risk assessment for a description risks associated with this
location.
Internal System Risk
· Internal system risks include software viruses and spam spread either intentionally or
unintentionally throughout the network; viruses in particular can render the network unusable.
· Viruses: the vast majority of current viruses are transmitted via email. Viruses cause a
reduction in productivity on workstations, and require a Data Processing technician to clean or
re-image the computer.
· Incoming spam: SPAM accounts for a significant amount of the emails our servers process, our
spam filters are currently filtering out over 95% of the spam emails received. Furthermore,
spam can introduce viruses and/or spyware onto a user's workstation.
· Outgoing spam: if the Town's network was to be used to relay spam out to the Internet, our
systems will likely be blacklisted, preventing our users from sending legitimate messages.
· Hardware failure: physical failure of the hardware in the server will cause downtime and may
cause data corruption.
· Data compromise via web applications: there are a number of different kinds of attacks on web
applications such as Outlook Web Access (OWA). They can allow an attacker to run programs
on the server, masquerade as the user, etc.
· System level compromise via various running services ("Remote" Compromise): a flaw in any
service running on a server could potentially be used to compromise the server by a remote
attacker unless additional measures are taken.
· System level compromise by a local user ("Local" Compromise): local users are those users
that actually have an account on the server. By necessity, they have additional rights above
those given to an anonymous user.
· Accidental misconflguration by an administrator: the system administrator, by necessity, has
the ability to make drastic changes to server functionality. These changes can cause major
problems to the functionality of the server, if not done properly. Should the administrator that
made the change not be available to correct the problem, the alternate administrator can have
difficulties determining what changes were made and how to restore previous functionality.
· Passwords passed in the clear; most email services transfer a user's password in clear text (via
HTTP). This allows a malicious user to easily read the user's password and then masquerade as
the user to send and receive messages as that user.
External System Risk
· Town email services are dependent upon the servers and network for continued operation. This
includes Lightpath and Cablevision connectivity.
Recovery Planning
· General Recovery Steps:
o Assess which network service or services have been lost.
Notify the user base, by whatever means are available, as to the service outage.
Troubleshoot to isolate the cause of the service outage.
Once the problem is isolated, take appropriate action to restore the service(s).
In the event the service cannot be restored in a timely fashion, assess possible
workarounds, including temporary outsourcing, if feasible.
Notify the user base as to the status of the affected service.
Notify the user base when the service becomes available.
In the case of a major virus infection, ensure that the virus protection software is
updated and establish a Data Processing disaster team to clean infected computers.
Restore email system state and any possible data.
For hardware failure, acquire replacement parts as needed and/or contact the appropriate
maintenance contract vendor.
In the case of malicious activity by a single user, disconnect the server from the network
and determine the method of data corruption and the duration of inappropriate access.
Secure the system as necessary, likely including a full system restore, followed by
patching of the security flaw.
Preventative Measures
· Current preventative measures include:
Physical access to the servers -The door to the server room is kept locked at all times.
Viruses -The Town uses Computer Associates Threat Management for all servers and
workstations. The virus definitions are updated daily.
~ Incoming Spam - The Town is currently using GFI Mail Essentials and SpamAssassin as
the email spam blockers to drop email delivery attempts from known spam/virus
sources.
Outgoing Spam - The Town currently allows email relaying from its own IP address
blocks. This prevents a remote spammer from using the Town's mail servers directly.
Data Processing Team Members
Lloyd H. Reisenberg, Network and Systems Administrator
Zachary Tomaszewski, Technical Coordinator II