services. Overall goals (like assuring the integrity of critical
systems) might be the reason for not analyzing an incident. Of
course, this is an important management decision; but all involved
parties must be aware that without analysis the same incident may
happen again.
It is also important to prioritize the actions to be taken during an
incident well in advance of the time an incident occurs. Sometimes
an incident may be so complex that it is impossible to do everything
at once to respond to it; priorities are essential. Although
priorities will vary from institution to institution, the following
suggested priorities may serve as a starting point for defining your
organization's response:
(1) Priority one -- protect human life and people's
safety; human life always has precedence over all
other considerations.
(2) Priority two -- protect classified and/or sensitive
data. Prevent exploitation of classified and/or
sensitive systems, networks or sites. Inform affected
classified and/or sensitive systems, networks or sites
about already occurred penetrations.
(Be aware of regulations by your site or by government)
(3) Priority three -- protect other data, including
proprietary, scientific, managerial and other data,
because loss of data is costly in terms of resources.
Prevent exploitations of other systems, networks or
sites and inform already affected systems, networks or
sites about successful penetrations.
(4) Priority four -- prevent damage to systems (e.g., loss
or alteration of system files, damage to disk drives,
etc.). Damage to systems can result in costly down
time and recovery.
(5) Priority five -- minimize disruption of computing
resources (including processes). It is better in many
cases to shut a system down or disconnect from a network
than to risk damage to data or systems. Sites will have
to evaluate the trade-offs between shutting down and
disconnecting, and staying up. There may be service
agreements in place that may require keeping systems
up even in light of further damage occurring. However,
the damage and scope of an incident may be so extensive
that service agreements may have to be over-ridden.
An important implication for defining priorities is that once human
life and national security considerations have been addressed, it is
generally more important to save data than system software and
hardware. Although it is undesirable to have any damage or loss
during an incident, systems can be replaced. However, the loss or
compromise of data (especially classified or proprietary data) is
usually not an acceptable outcome under any circumstances.
Another important concern is the effect on others, beyond the systems
and networks where the incident occurs. Within the limits imposed by
government regulations it is always important to inform affected
parties as soon as possible. Due to the legal implications of this
topic, it should be included in the planned procedures to avoid
further delays and uncertainties for the administrators.
Any plan for responding to security incidents should be guided by
local policies and regulations. Government and private sites that
deal with classified material have specific rules that they must
follow.
The policies chosen by your site on how it reacts to incidents will
shape your response. For example, it may make little sense to create
mechanisms to monitor and trace intruders if your site does not plan
to take action against the intruders if they are caught. Other
organizations may have policies that affect your plans. Telephone
companies often release information about telephone traces only to
law enforcement agencies.
Handling incidents can be tedious and require any number of routine
tasks that could be handled by support personnel. To free the
technical staff it may be helpful to identify support staff who will
help with tasks like: photocopying, fax'ing, etc.
5.2 Notification and Points of Contact
It is important to establish contacts with various personnel before a
real incident occurs. Many times, incidents are not real
emergencies. Indeed, often you will be able to handle the activities
internally. However, there will also be many times when others
outside your immediate department will need to be included in the
incident handling. These additional contacts include local managers
and system administrators, administrative contacts for other sites on
the Internet, and various investigative organizations. Getting to
know these contacts before incidents occurs will help to make your
incident handling process more efficient.
For each type of communication contact, specific "Points of Contact"
(POC) should be defined. These may be technical or administrative in
nature and may include legal or investigative agencies as well as
service providers and vendors. When establishing these contact, it
is important to decide how much information will be shared with each
class of contact. It is especially important to define, ahead of
time, what information will be shared with the users at a site, with
the public (including the press), and with other sites.
Settling these issues are especially important for the local person
responsible for handling the incident, since that is the person
responsible for the actual notification of others. A list of
contacts in each of these categories is an important time saver for
this person during an incident. It can be quite difficult to find an
appropriate person during an incident when many urgent events are
ongoing. It is strongly recommended that all relevant telephone
numbers (also electronic mail addresses and fax numbers) be included
in the site security policy. The names and contact information of
all individuals who will be directly involved in the handling of an
incident should be placed at the top of this list.
5.2.1 Local Managers and Personnel
When an incident is under way, a major issue is deciding who is in
charge of coordinating the activity of the multitude of players. A
major mistake that can be made is to have a number of people who are
each working independently, but are not working together. This will
only add to the confusion of the event and will probably lead to
wasted or ineffective effort.
The single POC may or may not be the person responsible for handling
the incident. There are two distinct roles to fill when deciding who
shall be the POC and who will be the person in charge of the
incident. The person in charge of the incident will make decisions
as to the interpretation of policy applied to the event. In
contrast, the POC must coordinate the effort of all the parties
involved with handling the event.
The POC must be a person with the technical expertise to successfully
coordinate the efforts of the system managers and users involved in
monitoring and reacting to the attack. Care should be taken when
identifying who this person will be. It should not necessarily be
the same person who has administrative responsibility for the
compromised systems since often such administrators have knowledge
only sufficient for the day to day use of the computers, and lack in
depth technical expertise.
Another important function of the POC is to maintain contact with law
enforcement and other external agencies to assure that multi-agency
involvement occurs. The level of involvement will be determined by
management decisions as well as legal constraints.
A single POC should also be the single person in charge of collecting
evidence, since as a rule of thumb, the more people that touch a
potential piece of evidence, the greater the possibility that it will
be inadmissible in court. To ensure that evidence will be acceptable
to the legal community, collecting evidence should be done following
predefined procedures in accordance with local laws and legal
regulations.
One of the most critical tasks for the POC is the coordination of all
relevant processes. Responsibilities may be distributed over the
whole site, involving multiple independent departments or groups.
This will require a well coordinated effort in order to achieve
overall success. The situation becomes even more complex if multiple
sites are involved. When this happens, rarely will a single POC at
one site be able to adequately coordinate the handling of the entire
incident. Instead, appropriate incident response teams should be
involved.
The incident handling process should provide some escalation
mechanisms. In order to define such a mechanism, sites will need to
create an internal classification scheme for incidents. Associated
with each level of incident will be the appropriate POC and
procedures. As an incident is escalated, there may be a change in
the POC which will need to be communicated to all others involved in
handling the incident. When a change in the POC occurs, old POC
should brief the new POC in all background information.
Lastly, users must know how to report suspected incidents. Sites
should establish reporting procedures that will work both during and
outside normal working hours. Help desks are often used to receive
these reports during normal working hours, while beepers and
telephones can be used for out of hours reporting.
5.2.2 Law Enforcement and Investigative Agencies
In the event of an incident that has legal consequences, it is
important to establish contact with investigative agencies (e.g, the
FBI and Secret Service in the U.S.) as soon as possible. Local law
enforcement, local security offices, and campus police departments
should also be informed as appropriate. This section describes many
of the issues that will be confronted, but it is acknowledged that
each organization will have its own local and governmental laws and
regulations that will impact how they interact with law enforcement
and investigative agencies. The most important point to make is that
each site needs to work through these issues.
A primary reason for determining these point of contact well in
advance of an incident is that once a major attack is in progress,
there is little time to call these agencies to determine exactly who
the correct point of contact is. Another reason is that it is
important to cooperate with these agencies in a manner that will
foster a good working relationship, and that will be in accordance
with the working procedures of these agencies. Knowing the working
procedures in advance, and the expectations of your point of contact
is a big step in this direction. For example, it is important to
gather evidence that will be admissible in any subsequent legal
proceedings, and this will require prior knowledge of how to gather
such evidence. A final reason for establishing contacts as soon as
possible is that it is impossible to know the particular agency that
will assume jurisdiction in any given incident. Making contacts and
finding the proper channels early on will make responding to an
incident go considerably more smoothly.
If your organization or site has a legal counsel, you need to notify
this office soon after you learn that an incident is in progress. At
a minimum, your legal counsel needs to be involved to protect the
legal and financial interests of your site or organization. There
are many legal and practical issues, a few of which are:
(1) Whether your site or organization is willing to risk negative
publicity or exposure to cooperate with legal prosecution
efforts.
(2) Downstream liability--if you leave a compromised system as is so
it can be monitored and another computer is damaged because the
attack originated from your system, your site or organization
may be liable for damages incurred.
(3) Distribution of information--if your site or organization
distributes information about an attack in which another site or
organization may be involved or the vulnerability in a product
that may affect ability to market that product, your site or
organization may again be liable for any damages (including
damage of reputation).
(4) Liabilities due to monitoring--your site or organization may be
sued if users at your site or elsewhere discover that your site
is monitoring account activity without informing users.
Unfortunately, there are no clear precedents yet on the liabilities
or responsibilities of organizations involved in a security incident
or who might be involved in supporting an investigative effort.
Investigators will often encourage organizations to help trace and
monitor intruders. Indeed, most investigators cannot pursue computer
intrusions without extensive support from the organizations involved.
However, investigators cannot provide protection from liability
claims, and these kinds of efforts may drag out for months and may
take a lot of effort.
On the other hand, an organization's legal council may advise extreme
caution and suggest that tracing activities be halted and an intruder
shut out of the system. This, in itself, may not provide protection
from liability, and may prevent investigators from identifying the
perpetrator.
The balance between supporting investigative activity and limiting
liability is tricky. You'll need to consider the advice of your legal
counsel and the damage the intruder is causing (if any) when making
your decision about what to do during any particular incident.
Your legal counsel should also be involved in any decision to contact
investigative agencies when an incident occurs at your site. The
decision to coordinate efforts with investigative agencies is most
properly that of your site or organization. Involving your legal
counsel will also foster the multi-level coordination between your
site and the particular investigative agency involved, which in turn
results in an efficient division of labor. Another result is that
you are likely to obtain guidance that will help you avoid future
legal mistakes.
Finally, your legal counsel should evaluate your site's written
procedures for responding to incidents. It is essential to obtain a
"clean bill of health" from a legal perspective before you actually
carry out these procedures.
It is vital, when dealing with investigative agencies, to verify that
the person who calls asking for information is a legitimate
representative from the agency in question. Unfortunately, many well
intentioned people have unknowingly leaked sensitive details about
incidents, allowed unauthorized people into their systems, etc.,
because a caller has masqueraded as a representative of a government
agency. (Note: this word of caution actually applies to all external
contacts.)
A similar consideration is using a secure means of communication.
Because many network attackers can easily re-route electronic mail,
avoid using electronic mail to communicate with other agencies (as
well as others dealing with the incident at hand). Non-secured phone
lines (the phones normally used in the business world) are also
frequent targets for tapping by network intruders, so be careful!
There is no one established set of rules for responding to an
incident when the local government becomes involved. Normally (in
the U.S.), except by legal order, no agency can force you to monitor,
to disconnect from the network, to avoid telephone contact with the
suspected attackers, etc. Each organization will have a set of local
and national laws and regulations that must be adhered to when
handling incidents. It is recommended that each site be familiar with
those laws and regulations, and identify and get know the contacts
for agencies with jurisdiction well in advance of handling an
incident.
5.2.3 Computer Security Incident Handling Teams
There are currently a number of of Computer Security Incident
Response teams (CSIRTs) such as the CERT Coordination Center, the
German DFN-CERT, and other teams around the globe. Teams exist for
many major government agencies and large corporations. If such a
team is available, notifying it should be of primary consideration
during the early stages of an incident. These teams are responsible
for coordinating computer security incidents over a range of sites
and larger entities. Even if the incident is believed to be
contained within a single site, it is possible that the information
available through a response team could help in fully resolving the
incident.
If it is determined that the breach occurred due to a flaw in the
system's hardware or software, the vendor (or supplier) and a
Computer Security Incident Handling team should be notified as soon
as possible. This is especially important because many other systems
are vulnerable, and these vendor and response team organizations can
help disseminate help to other affected sites.
In setting up a site policy for incident handling, it may be
desirable to create a subgroup, much like those teams that already
exist, that will be responsible for handling computer security
incidents for the site (or organization). If such a team is created,
it is essential that communication lines be opened between this team
and other teams. Once an incident is under way, it is difficult to
open a trusted dialogue between other teams if none has existed
before.
5.2.4 Affected and Involved Sites
If an incident has an impact on other sites, it is good practice to
inform them. It may be obvious from the beginning that the incident
is not limited to the local site, or it may emerge only after further
analysis.
Each site may choose to contact other sites directly or they can pass
the information to an appropriate incident response team. It is often
very difficult to find the responsible POC at remote sites and the
incident response team will be able to facilitate contact by making
use of already established channels.
The legal and liability issues arising from a security incident will
differ from site to site. It is important to define a policy for the
sharing and logging of information about other sites before an
incident occurs.
Information about specific people is especially sensitive, and may be
subject to privacy laws. To avoid problems in this area, irrelevant
information should be deleted and a statement of how to handle the
remaining information should be included. A clear statement of how
this information is to be used is essential. No one who informs a
site of a security incident wants to read about it in the public
press. Incident response teams are valuable in this respect. When
they pass information to responsible POCs, they are able to protect
the anonymity of the original source. But, be aware that, in many
cases, the analysis of logs and information at other sites will
reveal addresses of your site.
All the problems discussed above should be not taken as reasons not
to involve other sites. In fact, the experiences of existing teams
reveal that most sites informed about security problems are not even
aware that their site had been compromised. Without timely
information, other sites are often unable to take action against
intruders.
5.2.5 Internal Communications
It is crucial during a major incident to communicate why certain
actions are being taken, and how the users (or departments) are
expected to behave. In particular, it should be made very clear to
users what they are allowed to say (and not say) to the outside world
(including other departments). For example, it wouldn't be good for
an organization if users replied to customers with something like,
"I'm sorry the systems are down, we've had an intruder and we are
trying to clean things up." It would be much better if they were
instructed to respond with a prepared statement like, "I'm sorry our
systems are unavailable, they are being maintained for better service
in the future."
Communications with customers and contract partners should be handled
in a sensible, but sensitive way. One can prepare for the main issues
by preparing a checklist. When an incident occurs, the checklist can
be used with the addition of a sentence or two for the specific
circumstances of the incident.
Public relations departments can be very helpful during incidents.
They should be involved in all planning and can provide well
constructed responses for use when contact with outside departments
and organizations is necessary.
5.2.6 Public Relations - Press Releases
There has been a tremendous growth in the amount of media coverage
dedicated to computer security incidents in the United States. Such
press coverage is bound to extend to other countries as the Internet
continues to grow and expand internationally. Readers from countries
where such media attention has not yet occurred, can learn from the
experiences in the U.S. and should be forwarned and prepared.
One of the most important issues to consider is when, who, and how
much to release to the general public through the press. There are
many issues to consider when deciding this particular issue. First
and foremost, if a public relations office exists for the site, it is
important to use this office as liaison to the press. The public
relations office is trained in the type and wording of information
released, and will help to assure that the image of the site is
protected during and after the incident (if possible). A public
relations office has the advantage that you can communicate candidly
with them, and provide a buffer between the constant press attention
and the need of the POC to maintain control over the incident.
If a public relations office is not available, the information
released to the press must be carefully considered. If the
information is sensitive, it may be advantageous to provide only
minimal or overview information to the press. It is quite possible
that any information provided to the press will be quickly reviewed
by the perpetrator of the incident. Also note that misleading the
press can often backfire and cause more damage than releasing
sensitive information.
While it is difficult to determine in advance what level of detail to
provide to the press, some guidelines to keep in mind are:
(1) Keep the technical level of detail low. Detailed
information about the incident may provide enough
information for others to launch similar attacks on
other sites, or even damage the site's ability to
prosecute the guilty party once the event is over.
(2) Keep the speculation out of press statements.
Speculation of who is causing the incident or the
motives are very likely to be in error and may cause
an inflamed view of the incident.
(3) Work with law enforcement professionals to assure that
evidence is protected. If prosecution is involved,
assure that the evidence collected is not divulged to
the press.
(4) Try not to be forced into a press interview before you are
prepared. The popular press is famous for the "2 am"
interview, where the hope is to catch the interviewee off
guard and obtain information otherwise not available.
(5) Do not allow the press attention to detract from the
handling of the event. Always remember that the successful
closure of an incident is of primary importance.
5.3 Identifying an Incident
5.3.1 Is It Real?
This stage involves determining if a problem really exists. Of
course many if not most signs often associated with virus infection,
system intrusions, malicious users, etc., are simply anomalies such
as hardware failures or suspicious system/user behavior. To assist
in identifying whether there really is an incident, it is usually
helpful to obtain and use any detection software which may be
available. Audit information is also extremely useful, especially in
determining whether there is a network attack. It is extremely
important to obtain a system snapshot as soon as one suspects that
something is wrong. Many incidents cause a dynamic chain of events
to occur, and an initial system snapshot may be the most valuable
tool for identifying the problem and any source of attack. Finally,
it is important to start a log book. Recording system events,
telephone conversations, time stamps, etc., can lead to a more rapid
and systematic identification of the problem, and is the basis for
subsequent stages of incident handling.
There are certain indications or "symptoms" of an incident that
deserve special attention:
(1) System crashes.
(2) New user accounts (the account RUMPLESTILTSKIN has been
unexpectedly created), or high activity on a previously
low usage account.
(3) New files (usually with novel or strange file names,
such as data.xx or k or .xx ).
(4) Accounting discrepancies (in a UNIX system you might
notice the shrinking of an accounting file called
/usr/admin/lastlog, something that should make you very
suspicious that there may be an intruder).
(5) Changes in file lengths or dates (a user should be
suspicious if .EXE files in an MS DOS computer have
unexplainedly grown by over 1800 bytes).
(6) Attempts to write to system (a system manager notices
that a privileged user in a VMS system is attempting to
alter RIGHTSLIST.DAT).
(7) Data modification or deletion (files start to disappear).
(8) Denial of service (a system manager and all other users
become locked out of a UNIX system, now in single user mode).
(9) Unexplained, poor system performance
(10) Anomalies ("GOTCHA" is displayed on the console or there
are frequent unexplained "beeps").
(11) Suspicious probes (there are numerous unsuccessful login
attempts from another node).
(12) Suspicious browsing (someone becomes a root user on a UNIX
system and accesses file after file on many user accounts.)
(13) Inability of a user to log in due to modifications of his/her
account.
By no means is this list comprehensive; we have just listed a number
of common indicators. It is best to collaborate with other technical
and computer security personnel to make a decision as a group about
whether an incident is occurring.
5.3.2 Types and Scope of Incidents
Along with the identification of the incident is the evaluation of
the scope and impact of the problem. It is important to correctly
identify the boundaries of the incident in order to effectively deal
with it and prioritize responses.
In order to identify the scope and impact a set of criteria should be
defined which is appropriate to the site and to the type of
connections available. Some of the issues include:
(1) Is this a multi-site incident?
(2) Are many computers at your site affected by this incident?
(3) Is sensitive information involved?
(4) What is the entry point of the incident (network,
phone line, local terminal, etc.)?
(5) Is the press involved?
(6) What is the potential damage of the incident?
(7) What is the estimated time to close out the incident?
(8) What resources could be required to handle the incident?
(9) Is law enforcement involved?
5.3.3 Assessing the Damage and Extent
The analysis of the damage and extent of the incident can be quite
time consuming, but should lead to some insight into the nature of
the incident, and aid investigation and prosecution. As soon as the
breach has occurred, the entire system and all of its components
should be considered suspect. System software is the most probable
target. Preparation is key to be able to detect all changes for a
possibly tainted system. This includes checksumming all media from
the vendor using a algorithm which is resistant to tampering. (See
sections 4.3)
Assuming original vendor distribution media are available, an
analysis of all system files should commence, and any irregularities
should be noted and referred to all parties involved in handling the
incident. It can be very difficult, in some cases, to decide which
backup media are showing a correct system status. Consider, for
example, that the incident may have continued for months or years
before discovery, and the suspect may be an employee of the site, or
otherwise have intimate knowledge or access to the systems. In all
cases, the pre-incident preparation will determine what recovery is
possible.
If the system supports centralized logging (most do), go back over
the logs and look for abnormalities. If process accounting and
connect time accounting is enabled, look for patterns of system
usage. To a lesser extent, disk usage may shed light on the
incident. Accounting can provide much helpful information in an
analysis of an incident and subsequent prosecution. Your ability to
address all aspects of a specific incident strongly depends on the
success of this analysis.
5.4 Handling an Incident
Certain steps are necessary to take during the handling of an
incident. In all security related activities, the most important
point to be made is that all sites should have policies in place.
Without defined policies and goals, activities undertaken will remain
without focus. The goals should be defined by management and legal
counsel in advance.
One of the most fundamental objectives is to restore control of the
affected systems and to limit the impact and damage. In the worst
case scenario, shutting down the system, or disconnecting the system
from the network, may the only practical solution.
As the activities involved are complex, try to get as much help as
necessary. While trying to solve the problem alone, real damage
might occur due to delays or missing information. Most
administrators take the discovery of an intruder as a personal
challenge. By proceeding this way, other objectives as outlined in
the local policies may not always be considered. Trying to catch
intruders may be a very low priority, compared to system integrity,
for example. Monitoring a hacker's activity is useful, but it might
not be considered worth the risk to allow the continued access.
5.4.1 Types of Notification and Exchange of Information
When you have confirmed that an incident is occurring, the
appropriate personnel must be notified. How this notification is
achieved is very important to keeping the event under control both
from a technical and emotional standpoint. The circumstances should
be described in as much detail as possible, in order to aid prompt
acknowledgment and understanding of the problem. Great care should
be taken when determining to which groups detailed technical
information is given during the notification. For example, it is
helpful to pass this kind of information to an incident handling team
as they can assist you by providing helpful hints for eradicating the
vulnerabilities involved in an incident. On the other hand, putting
the critical knowledge into the public domain (e.g., via USENET
newsgroups or mailing lists) may potentially put a large number of
systems at risk of intrusion. It is invalid to assume that all
administrators reading a particular newsgroup have access to
operating system source code, or can even understand an advisory well
enough to take adequate steps.
First of all, any notification to either local or off-site personnel
must be explicit. This requires that any statement (be it an
electronic mail message, phone call, fax, beeper, or semaphone)
providing information about the incident be clear, concise, and fully
qualified. When you are notifying others that will help you handle
an event, a "smoke screen" will only divide the effort and create
confusion. If a division of labor is suggested, it is helpful to
provide information to each participant about what is being
accomplished in other efforts. This will not only reduce duplication
of effort, but allow people working on parts of the problem to know
where to obtain information relevant to their part of the incident.
Another important consideration when communicating about the incident
is to be factual. Attempting to hide aspects of the incident by
providing false or incomplete information may not only prevent a
successful resolution to the incident, but may even worsen the
situation.
The choice of language used when notifying people about the incident
can have a profound effect on the way that information is received.
When you use emotional or inflammatory terms, you raise the potential
for damage and negative outcomes of the incident. It is important to
remain calm both in written and spoken communications.
Another consideration is that not all people speak the same language.
Due to this fact, misunderstandings and delay may arise, especially
if it is a multi-national incident. Other international concerns
include differing legal implications of a security incident and
cultural differences. However, cultural differences do not only
exist between countries. They even exist within countries, between
different social or user groups. For example, an administrator of a
university system might be very relaxed about attempts to connect to
the system via telnet, but the administrator of a military system is
likely to consider the same action as a possible attack.
Another issue associated with the choice of language is the
notification of non-technical or off-site personnel. It is important
to accurately describe the incident without generating undue alarm or
confusion. While it is more difficult to describe the incident to a
non-technical audience, it is often more important. A non-technical
description may be required for upper-level management, the press, or
law enforcement liaisons. The importance of these communications
cannot be underestimated and may make the difference between
resolving the incident properly and escalating to some higher level
of damage.
If an incident response team becomes involved, it might be necessary
to fill out a template for the information exchange. Although this
may seem to be an additional burden and adds a certain delay, it
helps the team to act on this minimum set of information. The
response team may be able to respond to aspects of the incident of
which the local administrator is unaware. If information is given out
to someone else, the following minimum information should be
provided:
(1) timezone of logs, ... in GMT or local time
(2) information about the remote system, including host names,
IP addresses and (perhaps) user IDs
(3) all log entries relevant for the remote site
(4) type of incident (what happened, why should you care)
If local information (i.e., local user IDs) is included in the log
entries, it will be necessary to sanitize the entries beforehand to
avoid privacy issues. In general, all information which might assist
a remote site in resolving an incident should be given out, unless
local policies prohibit this.
5.4.2 Protecting Evidence and Activity Logs
When you respond to an incident, document all details related to the
incident. This will provide valuable information to yourself and
others as you try to unravel the course of events. Documenting all
details will ultimately save you time. If you don't document every
relevant phone call, for example, you are likely to forget a
significant portion of information you obtain, requiring you to
contact the source of information again. At the same time, recording
details will provide evidence for prosecution efforts, providing the
case moves in that direction. Documenting an incident will also help
you perform a final assessment of damage (something your management,
as well as law enforcement officers, will want to know), and will
provide the basis for later phases of the handling process:
eradication, recovery, and follow-up "lessons learned."
During the initial stages of an incident, it is often infeasible to
determine whether prosecution is viable, so you should document as if
you are gathering evidence for a court case. At a minimum, you
should record:
(1) all system events (audit records)
(2) all actions you take (time tagged)
(3) all external conversations (including the person with whom
you talked, the date and time, and the content of the
conversation)
The most straightforward way to maintain documentation is keeping a
log book. This allows you to go to a centralized, chronological
source of information when you need it, instead of requiring you to
page through individual sheets of paper. Much of this information is
potential evidence in a court of law. Thus, when a legal follow-up
is a possibility, one should follow the prepared procedures and avoid
jeopardizing the legal follow-up by improper handling of possible
evidence. If appropriate, the following steps may be taken.
(1) Regularly (e.g., every day) turn in photocopied, signed
copies of your logbook (as well as media you use to record
system events) to a document custodian.
(2) The custodian should store these copied pages in a secure
place (e.g., a safe).
(3) When you submit information for storage, you should
receive a signed, dated receipt from the document
custodian.
Failure to observe these procedures can result in invalidation of any
evidence you obtain in a court of law.
5.4.3 Containment
The purpose of containment is to limit the extent of an attack. An
essential part of containment is decision making (e.g., determining
whether to shut a system down, disconnect from a network, monitor
system or network activity, set traps, disable functions such as
remote file transfer, etc.).
Sometimes this decision is trivial; shut the system down if the
information is classified, sensitive, or proprietary. Bear in mind
that removing all access while an incident is in progress obviously
notifies all users, including the alleged problem users, that the
administrators are aware of a problem; this may have a deleterious
effect on an investigation. In some cases, it is prudent to remove
all access or functionality as soon as possible, then restore normal
operation in limited stages. In other cases, it is worthwhile to
risk some damage to the system if keeping the system up might enable
you to identify an intruder.
This stage should involve carrying out predetermined procedures.
Your organization or site should, for example, define acceptable
risks in dealing with an incident, and should prescribe specific
actions and strategies accordingly. This is especially important
when a quick decision is necessary and it is not possible to first
contact all involved parties to discuss the decision. In the absence
of predefined procedures, the person in charge of the incident will
often not have the power to make difficult management decisions (like
to lose the results of a costly experiment by shutting down a
system). A final activity that should occur during this stage of
incident handling is the notification of appropriate authorities.
5.4.4 Eradication
Once the incident has been contained, it is time to eradicate the
cause. But before eradicating the cause, great care should be taken
to collect all necessary information about the compromised system(s)
and the cause of the incident as they will likely be lost when
cleaning up the system.
Software may be available to help you in the eradication process,
such as anti-virus software. If any bogus files have been created,
archive them before deleting them. In the case of virus infections,
it is important to clean and reformat any media containing infected
files. Finally, ensure that all backups are clean. Many systems
infected with viruses become periodically re-infected simply because
people do not systematically eradicate the virus from backups. After
eradication, a new backup should be taken.
Removing all vulnerabilities once an incident has occurred is
difficult. The key to removing vulnerabilities is knowledge and
understanding of the breach.
It may be necessary to go back to the original distribution media and
re-customize the system. To facilitate this worst case scenario, a
record of the original system setup and each customization change
should be maintained. In the case of a network-based attack, it is
important to install patches for each operating system vulnerability
which was exploited.
As discussed in section 5.4.2, a security log can be most valuable
during this phase of removing vulnerabilities. The logs showing how
the incident was discovered and contained can be used later to help
determine how extensive the damage was from a given incident. The
steps taken can be used in the future to make sure the problem does
not resurface. Ideally, one should automate and regularly apply the
same test as was used to detect the security incident.
If a particular vulnerability is isolated as having been exploited,
the next step is to find a mechanism to protect your system. The
security mailing lists and bulletins would be a good place to search
for this information, and you can get advice from incident response
teams.
5.4.5 Recovery
Once the cause of an incident has been eradicated, the recovery phase
defines the next stage of action. The goal of recovery is to return
the system to normal. In general, bringing up services in the order
of demand to allow a minimum of user inconvenience is the best
practice. Understand that the proper recovery procedures for the
system are extremely important and should be specific to the site.
5.4.6 Follow-Up
Once you believe that a system has been restored to a "safe" state,
it is still possible that holes, and even traps, could be lurking in
the system. One of the most important stages of responding to
incidents is also the most often omitted, the follow-up stage. In
the follow-up stage, the system should be monitored for items that
may have been missed during the cleanup stage. It would be prudent
to utilize some of the tools mentioned in chapter 7 as a start.
Remember, these tools don't replace continual system monitoring and
good systems administration practices.
The most important element of the follow-up stage is performing a
postmortem analysis. Exactly what happened, and at what times? How
well did the staff involved with the incident perform? What kind of
information did the staff need quickly, and how could they have
gotten that information as soon as possible? What would the staff do
differently next time?
After an incident, it is prudent to write a report describing the
exact sequence of events: the method of discovery, correction
procedure, monitoring procedure, and a summary of lesson learned.
This will aid in the clear understanding of the problem. Creating a
formal chronology of events (including time stamps) is also important
for legal reasons.
A follow-up report is valuable for many reasons. It provides a
reference to be used in case of other similar incidents. It is also
important to, as quickly as possible obtain a monetary estimate of
the amount of damage the incident caused. This estimate should
include costs associated with any loss of software and files
(especially the value of proprietary data that may have been
disclosed), hardware damage, and manpower costs to restore altered
files, reconfigure affected systems, and so forth. This estimate may
become the basis for subsequent prosecution activity. The report can
also help justify an organization's computer security effort to
management.
5.5 Aftermath of an Incident
In the wake of an incident, several actions should take place. These
actions can be summarized as follows:
(1) An inventory should be taken of the systems' assets,
(i.e., a careful examination should determine how the
system was affected by the incident).
(2) The lessons learned as a result of the incident
should be included in revised security plan to
prevent the incident from re-occurring.
(3) A new risk analysis should be developed in light of the
incident.
(4) An investigation and prosecution of the individuals
who caused the incident should commence, if it is
deemed desirable.
If an incident is based on poor policy, and unless the policy is
changed, then one is doomed to repeat the past. Once a site has
recovered from and incident, site policy and procedures should be
reviewed to encompass changes to prevent similar incidents. Even
without an incident, it would be prudent to review policies and
procedures on a regular basis. Reviews are imperative due to today's
changing computing environments.
The whole purpose of this post mortem process is to improve all
security measures to protect the site against future attacks. As a
result of an incident, a site or organization should gain practical
knowledge from the experience. A concrete goal of the post mortem is
to develop new proactive methods. Another important facet of the
aftermath may be end user and administrator education to prevent a
reoccurrence of the security problem.
5.6 Responsibilities
5.6.1 Not Crossing the Line
It is one thing to protect one's own network, but quite another to
assume that one should protect other networks. During the handling
of an incident, certain system vulnerabilities of one's own systems
and the systems of others become apparent. It is quite easy and may
even be tempting to pursue the intruders in order to track them.
Keep in mind that at a certain point it is possible to "cross the
line," and, with the best of intentions, become no better than the
intruder.
The best rule when it comes to propriety is to not use any facility
of remote sites which is not public. This clearly excludes any entry
onto a system (such as a remote shell or login session) which is not
expressly permitted. This may be very tempting; after a breach of
security is detected, a system administrator may have the means to
"follow it up," to ascertain what damage is being done to the remote
site. Don't do it! Instead, attempt to reach the appropriate point
of contact for the affected site.
5.6.2 Good Internet Citizenship
During a security incident there are two choices one can make.
First, a site can choose to watch the intruder in the hopes of
catching him; or, the site can go about cleaning up after the
incident and shut the intruder out of the systems. This is a
decision that must be made very thoughtfully, as there may be legal
liabilities if you choose to leave your site open, knowing that an
intruder is using your site as a launching pad to reach out to other
sites. Being a good Internet citizen means that you should try to
alert other sites that may have been impacted by the intruder. These
affected sites may be readily apparent after a thorough review of
your log files.
5.6.3 Administrative Response to Incidents
When a security incident involves a user, the site's security policy
should describe what action is to be taken. The transgression should
be taken seriously, but it is very important to be sure of the role
the user played. Was the user naive? Could there be a mistake in
attributing the security breach to the user? Applying administrative
action that assumes the user intentionally caused the incident may
not be appropriate for a user who simply made a mistake. It may be
appropriate to include sanctions more suitable for such a situation
in your policies (e.g., education or reprimand of a user) in addition
to more stern measures for intentional acts of intrusion and system
misuse.
6. Ongoing Activities
At this point in time, your site has hopefully developed a complete
security policy and has developed procedures to assist in the
configuration and management of your technology in support of those
policies. How nice it would be if you could sit back and relax at
this point and know that you were finished with the job of security.
Unfortunately, that isn't possible. Your systems and networks are
not a static environment, so you will need to review policies and
procedures on a regular basis. There are a number of steps you can
take to help you keep up with the changes around you so that you can
initiate corresponding actions to address those changes. The
following is a starter set and you may add others as appropriate for
your site.
(1) Subscribe to advisories that are issued by various security incident