Guidelines for Evidence Collection and Archiving
How a system administrator should gather and archive evidence after a security incident so that it stays trustworthy.
Brezinski, D. and T. Killalea, "Guidelines for Evidence Collection and Archiving", BCP 55, RFC 3227, DOI 10.17487/RFC3227, February 2002, <https://www.rfc-editor.org/info/rfc3227>.
@misc{rfc3227,
series = {Request for Comments},
number = 3227,
howpublished = {RFC 3227},
publisher = {RFC Editor},
doi = {10.17487/RFC3227},
url = {https://www.rfc-editor.org/info/rfc3227},
author = {D. Brezinski and T. Killalea},
title = {{Guidelines for Evidence Collection and Archiving}},
pagetotal = 10,
year = 2002,
month = feb,
} RFC text
Contents
Network Working Group D. Brezinski
Request for Comments: 3227 In-Q-Tel
BCP: 55 T. Killalea
Category: Best Current Practice neart.org
February 2002
Guidelines for Evidence Collection and Archiving
Status of this Memo
This document specifies an Internet Best Current Practices for the
Internet Community, and requests discussion and suggestions for
improvements. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2002). All Rights Reserved.
Abstract
A "security incident" as defined in the "Internet Security Glossary",
RFC 2828, is a security-relevant system event in which the system's
security policy is disobeyed or otherwise breached. The purpose of
this document is to provide System Administrators with guidelines on
the collection and archiving of evidence relevant to such a security
incident.
If evidence collection is done correctly, it is much more useful in
apprehending the attacker, and stands a much greater chance of being
admissible in the event of a prosecution.
Table of Contents
1 Introduction.................................................... 2
1.1 Conventions Used in this Document........................... 2
2 Guiding Principles during Evidence Collection................... 3
2.1 Order of Volatility......................................... 4
2.2 Things to avoid............................................. 4
2.3 Privacy Considerations...................................... 5
2.4 Legal Considerations........................................ 5
3 The Collection Procedure........................................ 6
3.1 Transparency................................................ 6
3.2 Collection Steps............................................ 6
4 The Archiving Procedure......................................... 7
4.1 Chain of Custody............................................ 7
4.2 The Archive................................................. 7
5 Tools you'll need............................................... 7
6 References...................................................... 8
7 Acknowledgements................................................ 8
8 Security Considerations......................................... 8
9 Authors' Addresses.............................................. 9
10 Full Copyright Statement.......................................10
1 Introduction
A "security incident" as defined in [RFC2828] is a security-relevant
system event in which the system's security policy is disobeyed or
otherwise breached. The purpose of this document is to provide
System Administrators with guidelines on the collection and archiving
of evidence relevant to such a security incident. It's not our
intention to insist that all System Administrators rigidly follow
these guidelines every time they have a security incident. Rather,
we want to provide guidance on what they should do if they elect to
collect and protect information relating to an intrusion.
Such collection represents a considerable effort on the part of the
System Administrator. Great progress has been made in recent years
to speed up the re-installation of the Operating System and to
facilitate the reversion of a system to a 'known' state, thus making
the 'easy option' even more attractive. Meanwhile little has been
done to provide easy ways of archiving evidence (the difficult
option). Further, increasing disk and memory capacities and the more
widespread use of stealth and cover-your-tracks tactics by attackers
have exacerbated the problem.
If evidence collection is done correctly, it is much more useful in
apprehending the attacker, and stands a much greater chance of being
admissible in the event of a prosecution.
You should use these guidelines as a basis for formulating your
site's evidence collection procedures, and should incorporate your
site's procedures into your Incident Handling documentation. The
guidelines in this document may not be appropriate under all
jurisdictions. Once you've formulated your site's evidence
collection procedures, you should have law enforcement for your
jurisdiction confirm that they're adequate.
1.1 Conventions Used in this Document
The key words "REQUIRED", "MUST", "MUST NOT", "SHOULD", "SHOULD NOT",
and "MAY" in this document are to be interpreted as described in "Key
words for use in RFCs to Indicate Requirement Levels" [RFC2119].
2 Guiding Principles during Evidence Collection
- Adhere to your site's Security Policy and engage the
appropriate Incident Handling and Law Enforcement personnel.
- Capture as accurate a picture of the system as possible.
- Keep detailed notes. These should include dates and times. If
possible generate an automatic transcript. (e.g., On Unix
systems the 'script' program can be used, however the output
file it generates should not be to media that is part of the
evidence). Notes and print-outs should be signed and dated.
- Note the difference between the system clock and UTC. For each
timestamp provided, indicate whether UTC or local time is used.
- Be prepared to testify (perhaps years later) outlining all
actions you took and at what times. Detailed notes will be
vital.
- Minimise changes to the data as you are collecting it. This is
not limited to content changes; you should avoid updating file
or directory access times.
- Remove external avenues for change.
- When confronted with a choice between collection and analysis
you should do collection first and analysis later.
- Though it hardly needs stating, your procedures should be
implementable. As with any aspect of an incident response
policy, procedures should be tested to ensure feasibility,
particularly in a crisis. If possible procedures should be
automated for reasons of speed and accuracy. Be methodical.
- For each device, a methodical approach should be adopted which
follows the guidelines laid down in your collection procedure.
Speed will often be critical so where there are a number of
devices requiring examination it may be appropriate to spread
the work among your team to collect the evidence in parallel.
However on a single given system collection should be done step
by step.
- Proceed from the volatile to the less volatile (see the Order
of Volatility below).
- You should make a bit-level copy of the system's media. If you
wish to do forensics analysis you should make a bit-level copy
of your evidence copy for that purpose, as your analysis will
almost certainly alter file access times. Avoid doing
forensics on the evidence copy.
2.1 Order of Volatility
When collecting evidence you should proceed from the volatile to the
less volatile. Here is an example order of volatility for a typical
system.
- registers, cache
- routing table, arp cache, process table, kernel statistics,
memory
- temporary file systems
- disk
- remote logging and monitoring data that is relevant to the
system in question
- physical configuration, network topology
- archival media
2.2 Things to avoid
It's all too easy to destroy evidence, however inadvertently.
- Don't shutdown until you've completed evidence collection.
Much evidence may be lost and the attacker may have altered the
startup/shutdown scripts/services to destroy evidence.
- Don't trust the programs on the system. Run your evidence
gathering programs from appropriately protected media (see
below).
- Don't run programs that modify the access time of all files on
the system (e.g., 'tar' or 'xcopy').
- When removing external avenues for change note that simply
disconnecting or filtering from the network may trigger
"deadman switches" that detect when they're off the net and
wipe evidence.
2.3 Privacy Considerations
- Respect the privacy rules and guidelines of your company and
your legal jurisdiction. In particular, make sure no
information collected along with the evidence you are searching
for is available to anyone who would not normally have access
to this information. This includes access to log files (which
may reveal patterns of user behaviour) as well as personal data
files.
- Do not intrude on people's privacy without strong
justification. In particular, do not collect information from
areas you do not normally have reason to access (such as
personal file stores) unless you have sufficient indication
that there is a real incident.
- Make sure you have the backing of your company's established
procedures in taking the steps you do to collect evidence of an
incident.
2.4 Legal Considerations
Computer evidence needs to be
- Admissible: It must conform to certain legal rules before it
can be put before a court.
- Authentic: It must be possible to positively tie evidentiary
material to the incident.
- Complete: It must tell the whole story and not just a
particular perspective.
- Reliable: There must be nothing about how the evidence was
collected and subsequently handled that casts doubt about its
authenticity and veracity.
- Believable: It must be readily believable and understandable
by a court.
3 The Collection Procedure
Your collection procedures should be as detailed as possible. As is
the case with your overall Incident Handling procedures, they should
be unambiguous, and should minimise the amount of decision-making
needed during the collection process.
3.1 Transparency
The methods used to collect evidence should be transparent and
reproducible. You should be prepared to reproduce precisely the
methods you used, and have those methods tested by independent
experts.
3.2 Collection Steps
- Where is the evidence? List what systems were involved in the
incident and from which evidence will be collected.
- Establish what is likely to be relevant and admissible. When
in doubt err on the side of collecting too much rather than not
enough.
- For each system, obtain the relevant order of volatility.
- Remove external avenues for change.
- Following the order of volatility, collect the evidence with
tools as discussed in Section 5.
- Record the extent of the system's clock drift.
- Question what else may be evidence as you work through the
collection steps.
- Document each step.
- Don't forget the people involved. Make notes of who was there
and what were they doing, what they observed and how they
reacted.
Where feasible you should consider generating checksums and
cryptographically signing the collected evidence, as this may make it
easier to preserve a strong chain of evidence. In doing so you must
not alter the evidence.
4 The Archiving Procedure
Evidence must be strictly secured. In addition, the Chain of Custody
needs to be clearly documented.
4.1 Chain of Custody
You should be able to clearly describe how the evidence was found,
how it was handled and everything that happened to it.
The following need to be documented
- Where, when, and by whom was the evidence discovered and
collected.
- Where, when and by whom was the evidence handled or examined.
- Who had custody of the evidence, during what period. How was
it stored.
- When the evidence changed custody, when and how did the
transfer occur (include shipping numbers, etc.).
4.2 Where and how to Archive
If possible commonly used media (rather than some obscure storage
media) should be used for archiving.
Access to evidence should be extremely restricted, and should be
clearly documented. It should be possible to detect unauthorised
access.
5 Tools you'll need
You should have the programs you need to do evidence collection and
forensics on read-only media (e.g., a CD). You should have prepared
such a set of tools for each of the Operating Systems that you manage
in advance of having to use it.
Your set of tools should include the following:
- a program for examining processes (e.g., 'ps').
- programs for examining system state (e.g., 'showrev',
'ifconfig', 'netstat', 'arp').
- a program for doing bit-to-bit copies (e.g., 'dd', 'SafeBack').
- programs for generating checksums and signatures (e.g.,
'sha1sum', a checksum-enabled 'dd', 'SafeBack', 'pgp').
- programs for generating core images and for examining them
(e.g., 'gcore', 'gdb').
- scripts to automate evidence collection (e.g., The Coroner's
Toolkit [FAR1999]).
The programs in your set of tools should be statically linked, and
should not require the use of any libraries other than those on the
read-only media. Even then, since modern rootkits may be installed
through loadable kernel modules, you should consider that your tools
might not be giving you a full picture of the system.
You should be prepared to testify to the authenticity and reliability
of the tools that you use.
6 References
[FAR1999] Farmer, D., and W Venema, "Computer Forensics Analysis
Class Handouts", http://www.fish.com/forensics/
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2196] Fraser, B., "Site Security Handbook", FYI 8, RFC 2196,
September 1997.
[RFC2350] Brownlee, N. and E. Guttman, "Expectations for Computer
Security Incident Response", FYI 8, RFC 2350, June 1998.
[RFC2828] Shirey, R., "Internet Security Glossary", FYI 36, RFC
2828, May 2000.
7 Acknowledgements
We gratefully acknowledge the constructive comments received from
Harald Alvestrand, Byron Collie, Barbara Y. Fraser, Gordon Lennox,
Andrew Rees, Steve Romig and Floyd Short.
8 Security Considerations
This entire document discuses security issues.
9 Authors' Addresses
Dominique Brezinski
In-Q-Tel
1000 Wilson Blvd., Ste. 2900
Arlington, VA 22209
USA
EMail: dbrezinski@In-Q-Tel.org
Tom Killalea
Lisi/n na Bro/n
Be/al A/tha na Muice
Co. Mhaigh Eo
IRELAND
Phone: +1 206 266-2196
EMail: tomk@neart.org
10. Full Copyright Statement
Copyright (C) The Internet Society (2002). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.
AI ELI5
On this page
§1Purpose and Scope
A security incident, as defined in [RFC2828], is a security-relevant system event in which the system's security policy is disobeyed or otherwise breached. This RFC gives System Administrators guidelines for collecting and archiving evidence of such an incident.
The authors do not insist that guidelines be followed rigidly every time. They are guidance for administrators who elect to collect and protect information about an intrusion. Collection is a considerable effort, and faster reinstallation and reversion to a known state make the easy option more attractive, while little has been done to make archiving evidence easier.
Sites should use the guidelines as a basis for their own procedures, include them in Incident Handling documentation, and have law enforcement for their jurisdiction confirm they are adequate. The guidelines may not suit all jurisdictions.
Cleaning up fast is easy, but saving clues takes work. This RFC explains how to save them well if you decide to.
§2Guiding Principles
The RFC lists principles for the whole collection effort. The main ones:
- Follow your site's Security Policy and involve Incident Handling and Law Enforcement personnel.
- Capture as accurate a picture of the system as possible, and minimise changes to data, including file and directory access times.
- Keep detailed, signed and dated notes, and say whether each timestamp is UTC or local time. Note the difference between the system clock and UTC.
- Collect first and analyse later. Proceed from the volatile to the less volatile.
- Make a bit-level copy of the media, and do any forensics on a copy of that copy, since analysis will almost certainly alter access times.
- Test procedures so they are implementable in a crisis, and automate them if possible. Work step by step on one system, but spread several devices across the team.
Administrators should be prepared to testify, perhaps years later, about every action and when it happened.
Write down everything you do and when, change as little as you can, and save the clues before you study them. Work on a copy of the copy.
§2.1Order of Volatility and Things to Avoid
The order of volatility says what to collect first. The RFC's example order for a typical system runs: registers and cache; routing table, arp cache, process table, kernel statistics and memory; temporary file systems; disk; remote logging and monitoring data; physical configuration and network topology; and archival media.
It is easy to destroy evidence inadvertently. The RFC warns against these:
- Shutting down before collection is complete, since evidence may be lost and the attacker may have altered startup and shutdown scripts.
- Trusting the programs on the system. Run collection tools from appropriately protected media.
- Running programs such as
tarorxcopythat modify the access time of all files. - Simply disconnecting or filtering the system from the network, which may trigger "deadman switches" that wipe evidence.
Ice cream melts faster than a rock, so grab the ice cream first. And do not switch off the lights or trust the burglar's tools.
- 1Registers, cache — Most volatile; collect first.
- 2Routing table, arp cache, process table, kernel statistics, memory — Live system state.
- 3Temporary file systems — Short-lived files.
- 4Disk — Persistent storage.
- 5Remote logging and monitoring data — Relevant to the system in question.
- 6Physical configuration, network topology — How the system is built and connected.
- 7Archival media — Least volatile; collect last.
§2.3Privacy and Legal Considerations
Collectors must respect the privacy rules of their company and legal jurisdiction. Nothing collected alongside the evidence, such as log files that may reveal user behaviour or personal data files, should be available to anyone who would not normally have access. Administrators should not intrude without strong justification, should not collect from areas such as personal file stores unless there is sufficient indication of a real incident, and should have the backing of established company procedures.
For legal purposes computer evidence needs to be admissible, authentic, complete, reliable and believable. In short, it must meet court rules, be tied to the incident, tell the whole story, be free of doubt about how it was handled, and be understandable to a court.
Do not snoop through people's private things unless there is a real reason, and make sure the clues you keep are the kind a judge would trust.
§3The Collection Procedure
Collection procedures should be as detailed and unambiguous as possible, minimising the decision-making needed in the moment. The methods should be transparent and reproducible, and testable by independent experts.
The listed steps: identify which systems hold evidence; decide what is likely to be relevant and admissible, erring on the side of collecting too much; get the order of volatility for each system; remove external avenues for change; collect in that order with the prepared tools; record the extent of the system's clock drift; question what else may be evidence; document each step; and note the people involved and what they observed.
Where feasible, generate checksums and cryptographically sign the collected evidence to help preserve a strong chain of evidence, without altering the evidence.
Make a clear checklist ahead of time, follow it step by step, write down who was there, and stamp each clue so you can show it was not changed.
§4Archiving and Chain of Custody
Evidence must be strictly secured, and the chain of custody must be clearly documented. It should be possible to describe how the evidence was found, how it was handled and everything that happened to it.
Document where, when and by whom evidence was discovered and collected; where, when and by whom it was handled or examined; who had custody and for what period, and how it was stored; and, whenever custody changed, when and how the transfer happened, including shipping numbers.
For storage, use commonly used media rather than obscure media if possible. Access should be extremely restricted and clearly documented, and unauthorised access should be detectable.
Keep a logbook of who held each clue and when, lock the clues up, and make sure you would notice if someone sneaked a look.
§5Tools You Need
Prepare the programs for evidence collection and forensics on read-only media such as a CD, for each Operating System you manage, before you need them. The set should include a program for examining processes (for example ps), programs for system state (for example ifconfig, netstat, arp), a bit-to-bit copy program (for example dd), programs for checksums and signatures (for example sha1sum, pgp), programs for core images (for example gcore, gdb), and scripts to automate collection such as The Coroner's Toolkit.
The tools should be statically linked and need no libraries other than those on the read-only media. Even so, since modern rootkits may be installed through loadable kernel modules, the tools might not give a full picture of the system. Administrators should be ready to testify to the authenticity and reliability of the tools they use.
Keep a toolbox on a disc that cannot be changed, so a burglar cannot tamper with your tools. Even then, remember a clever burglar might hide things from them.