Linux-Forensics-Checklist.md 9.1 KB
Newer Older
Heiko Reese's avatar
Heiko Reese committed
1
2
3
4
5
6
7
8
# KIT-CERT's Checklist for Linux Forensics

## Preliminary Considerations

Forensic investigations of computer hardware is usually divided in two phases:
online forensics (analysis of the running system) and offline forensics
(examination of the permanent storage).

Heiko Reese's avatar
   
Heiko Reese committed
9
10
This document's primary focus is the first phase (online forensics). We assume
that the reader has root access to the compromised machine.
Heiko Reese's avatar
Heiko Reese committed
11

heiko.reese's avatar
   
heiko.reese committed
12
## Find a proper place to store your findings
Heiko Reese's avatar
Heiko Reese committed
13
14
15
16

Every action that interacts with the storage subsystem can potentially destroy
evidence (both data and metadata). Mounting external storage changes the
contents of `/etc/mtab` and the timestamps of the containing directory `/etc`.
heiko.reese's avatar
   
heiko.reese committed
17
18
19
20
21
22
Merely looking at the file (`cat /etc/mtab`) changes the access time of `/etc`.

### Pushing data onto the network

You may push your findings directly onto the network, thus preventing/minimizing
changes to the local filesystems. This only works if the compromized machine is
Heiko Reese's avatar
   
Heiko Reese committed
23
still able to make outgoing connections to the destination server .
heiko.reese's avatar
   
heiko.reese committed
24
25

Open a listener on your server:
Heiko Reese's avatar
   
Heiko Reese committed
26
```sh
heiko.reese's avatar
   
heiko.reese committed
27
28
29
30
nc -l 6789 >> logfilename.txt
```

To send the standard output of a command, simply add this
Heiko Reese's avatar
   
Heiko Reese committed
31
```sh
heiko.reese's avatar
   
heiko.reese committed
32
33
34
 | nc -w 2 name_or_ip_of_server 6789
```

Heiko Reese's avatar
   
Heiko Reese committed
35
Encrypt all data in transition to prevent eavesdropping. Simply insert
Heiko Reese's avatar
   
Heiko Reese committed
36
[`openssl`](https://openssl.org/) into the toolchain:
Heiko Reese's avatar
   
Heiko Reese committed
37
```sh
Heiko Reese's avatar
   
Heiko Reese committed
38
39
nc -l 6789 | openssl enc -aes128 -d -k supersecretpw >> log.txt
```
Heiko Reese's avatar
   
Heiko Reese committed
40
```sh
Heiko Reese's avatar
   
Heiko Reese committed
41
42
43
 | openssl enc -aes128 -e -k supersecretpw | nc -w 2 name_or_ip_of_server 6789
```

Heiko Reese's avatar
   
Heiko Reese committed
44
45
Use [cryptcat](http://cryptcat.sourceforge.net) if it's available on the
compromised machine.
heiko.reese's avatar
   
heiko.reese committed
46

Heiko Reese's avatar
   
Heiko Reese committed
47
48
49
50
51
52
53
54
55
56
57
To copy files, use `cat`:

```
cat /usr/bin/rootkit_0.1 | nc …
```

Use `dd` to  transfer whole blockdevices:
```
dd if=/dev/sdx23 | nc…
```

Heiko Reese's avatar
   
Heiko Reese committed
58
59
60
61
Using [fuse sshfs](http://fuse.sourceforge.net/sshfs.html) is discouraged for
two reasons. First, it touches lots of files ($HOME/.ssh/*, /etc). And more
importantly: attackers often change the ssh binaries to intercept passwords.

Heiko Reese's avatar
   
Heiko Reese committed
62
63
64
The same problems apply to the `… | ssh user@host 'cat > /my/destination/file`
approach.

Heiko Reese's avatar
   
Heiko Reese committed
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
### Collecting data on local storage

If you decide to collect your findings locally, please refrain from using
existing storage of the compromised system. There are two viable options:
external storage like USB-sticks or memory-backed filesystem aka `tmpfs`.
Please save a listing of all mounts in all namespaces before mounting anything.

Check all the different mounts:
```
md5sum /proc/mounts /proc/*/mounts | sort | uniq -d -w 32
```

Get creative to solve this chicken-egg-problem! If you have copy/paste on your
console, simply `cat`the files and copy the aferwards. Don't use screen/tmux,
the touch lots of files. Check for empty pre-existing `tmpfs`-filesystems.

Find a proper location for the mountpoint and Mount your device:
```
mount -t tmpfs none /mnt
# or
mount /dev/sdx1 /mnt
```

Heiko Reese's avatar
   
Heiko Reese committed
88
89
## Collecting evidence

Heiko Reese's avatar
   
Heiko Reese committed
90
91
Collect evidence by saving potentielly interesting parts of the system state.
Start with the most volatile and work your way down:
Heiko Reese's avatar
   
Heiko Reese committed
92
93
94
95

1. network and connection state
1. process state
1. users
Heiko Reese's avatar
   
Heiko Reese committed
96
1. system state and configuration
Heiko Reese's avatar
   
Heiko Reese committed
97
98
99
100
101
102

The following commands assume that you are writing your findings to a local
storage and that your current working directory is set accordingly.

Some programs have rather unstable commandline parameters, please adjust
accordingly (if possible, use `--help` instead of the manpage to find out). You
Heiko Reese's avatar
   
Heiko Reese committed
103
104
105
106
107
108
109
can find the long versions (if applicable) as comments above every command.

Some modern Linux systems have SELinux enabled. Run `getenforce` to find out if
SELinux is enforcing, permissive, or disabled. If the state is enforcing, we
need to get selinux information when applicable. Most tools provide a switch
`-Z` for that. Such commands are marked with a special comment like
`# SELinux: add "-Z"`.
Heiko Reese's avatar
   
Heiko Reese committed
110
111
112
113
114
115
116
117
118
119

### Network state

Get state of existing connections and open sockets:
```sh
# --verbose --wide --extend --timers --program --numeric (--listening)
netstat -v -W -e -o -p -n     > netstat_vWeopn.txt
netstat -v -W -e -o -p -n -l  > netstat_vWeopnl.txt
# same without --numeric
netstat -v -W -e -o -p        > netstat_vWeop.txt
Heiko Reese's avatar
   
Heiko Reese committed
120
netstat -v -W -e -o -p -l     > netstat_vWeop.txt
Heiko Reese's avatar
   
Heiko Reese committed
121
122
```

Heiko Reese's avatar
   
Heiko Reese committed
123
Redo using `ss` if available:
Heiko Reese's avatar
   
Heiko Reese committed
124
125
126
127
128
129

```sh
# --options --extended --processes --info --numeric (--listening )
ss -o -e -p -i -n    > ss_oepin.txt
ss -o -e -p -i -n -l > ss_oepinl.txt
# same without --numeric
Heiko Reese's avatar
   
Heiko Reese committed
130
131
ss -o -e -p -i       > ss_oepi.txt
ss -o -e -p -i -l    > ss_oepil.txt
Heiko Reese's avatar
   
Heiko Reese committed
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
```

Dump arp cache:
```sh
arp -n > arp_n.txt
ip neigh show > ip_neigh_show.txt
```

Get routing-related stuff:
```sh
for i in link addr route rule neigh ntable tunnel tuntap maddr mroute mrule; do
    ip $i list > ip_${i}_l.txt;
done
```

Capture iptable's state:
```sh
# --verbose --numeric --exact --list --table
for t in filter nat mangle raw; do iptables -v -n -x -L -t > iptables_vnxL_t${t}.txt; done
for table in filter mangle raw; do ip6tables -n -t ${table} -L -v -x > ip6tables_nt_${table}.txt; done
for table in filter nat broute; do ebtables -L --Lmac2 --Lc -t ${table} > ebtables_L_Lmac_Lc_t_${table}.txt; done
```

Heiko Reese's avatar
   
Heiko Reese committed
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
### Process State

Save process table:

```sh
ps auxwwwe > ps_auxwwwe.txt
```

Formatting of certain columns seems to be broken in many versions of `ps`, so
we have to add the :xxxxx-postfixes to enforce wide columns. This is not meant
for human consumption:

```sh
ps wwwe -A -o pid,ppid,sess,tname,tpgid,comm,f,uid,euid,rgid,ruid,gid,egid,fgid,ouid,pgid,sgid,suid,supgid,suser,pidns,unit,label,time,lstart,lsession,seat,machine,ni,wchan,etime,%cpu,%mem,cgroup:65535,args:65535 > ps_dump_e.txt
ps www -A -o pid,ppid,sess,tname,tpgid,comm,f,uid,euid,rgid,ruid,gid,egid,fgid,ouid,pgid,sgid,suid,supgid,suser,pidns,unit,label,time,lstart,lsession,seat,machine,ni,wchan,etime,%cpu,%mem,cgroup:65535,args:65535 > ps_dump.txt
```

Something more human-readable:
```sh
pstree -a -l -p -u    > pstree_alpu.txt
pstree -a -l -p -u -Z > pstree_alpuZ.txt
```

Heiko Reese's avatar
   
Heiko Reese committed
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
`lsof` has the most unstable commandline interface. We're planning to include versions for specific linux distributions in the future…

```sh
lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt
```

```sh
# time pid creator limits
 in t p c l; do ipcs -a -${i} > ipcs_a_${i}.txt;done
```

Add this on systems that use [systemd](http://www.freedesktop.org/wiki/Software/systemd/):
```sh
systemctl status -l > systemctl_status_l.txt
```

### Users

```sh
last > last.txt
lastlog > lastlog.txt
who > who.txt
w > w.txt
```

Add this on systems that use systemd:
```sh
loginctl list-sessions > loginctl_list-sessions.txt
for s in $(loginctl list-sessions --no-legend | awk '{print $1}'); do loginctl show-session ${s} > loginctl_show-session_${s}.txt; done
for u in $(loginctl list-users --no-legend | awk '{print $1}'); do loginctl show-user ${u} > loginctl_show-user_${u}.txt; done
```

Heiko Reese's avatar
   
Heiko Reese committed
210
211
212
213
214
215
216
217
218
219
220
221
### System State and Configuration

```sh
dmesg > dmesg.txt
cat /proc/mounts > proc_mounts.txt
# or use the all-namespace-encompassing version
for p in $(md5sum /proc/mounts /proc/*/mounts | sort | uniq -d -w 32 | awk '{print $2}'); do cat $p > ${p////_}; done
cat /proc/mdstat > proc_mdstat.txt
lspci > lspci.txt
uname -a > uname_a.txt
uptime > uptime.txt
```
Heiko Reese's avatar
   
Heiko Reese committed
222

Heiko Reese's avatar
   
Heiko Reese committed
223
224
225
226
227
228
229
230
231
# Dumping suspicious processess

Have a closer look at the process list. Do this for every suspicious process:

Assign it to the `PID` variable for later usage:
```sh
# insert correct PID here!
export PID=12345
```
Heiko Reese's avatar
   
Heiko Reese committed
232

Heiko Reese's avatar
   
Heiko Reese committed
233
234
235
236
237
Stop the process:
```sh
kill -STOP ${PID}
```

Heiko Reese's avatar
   
Heiko Reese committed
238
239
240
241
242
243
244
245
246
247
248
249
250
251
TODO: Add cgroups-freezer-variant and discuss it (freezer blocks gdb/gcore).

Preserve original location of executable (plus a broken symlink of file was deleted) and the contents:
```sh
ls -l /proc/${PID}/ > proc_${PID}_ls_l.txt
cat /proc/${PID}/exe > proc_${PID}_exe
```

Create a coredump to preserve the process memory:
```sh
gdb -nh -batch -ex gcore -p ${PID}
```

We have not found a way to dump the cores directly into an unnamed pipe and out
252
253
into the net. Using FIFOs does not work because gdb needs to seek within the
file while writing it. Using a tmpfs might fails because some coredumps can get
Heiko Reese's avatar
   
Heiko Reese committed
254
255
256
257
258
259
260
261
262
263
264
265
266
pretty big. There are no widely available and stable compressing filesystems
available for linux at the time of writing.

Check for shared memory segments:
```sh
# look for /dev/shms
less /proc/${PID}/map
```

Save some more state information about the process. The available data in the
`/proc/$PID/` of the procfs varies between different kernel versions. Please
check `/proc/self` to what's available and adjust the next commandline
accordingly.
Heiko Reese's avatar
   
Heiko Reese committed
267

Heiko Reese's avatar
   
Heiko Reese committed
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
```sh
# use "tar -cf - /proc/… | …" to pipe the tarball to stdout
tar cf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,\
loginuid,maps,mountinfo,sched,schedstat,sessionid,smaps,stack,stat,statm,status,\
syscall,wchan}
```

Have a look at the open files. If the file has been deleted, ls will append
` (deleted)` to the destination filename. The contents can still be accessed
using the symlinks in `/proc/${PID}/fd`. This often happens with malware
written in interpreted languages like perl and python. Save all interesting
open files now: ```sh
ls -l /proc/${PID}/fd > proc_${PID}_fd.txt
# copy interesting open files, substitute MYFD with file descriptor number
MYFD=1234
cat /proc/${PID}/${MYFD}> proc_${PID}_fd_${MYFD}
```
Heiko Reese's avatar
   
Heiko Reese committed
285
286
287
288


#### Authors:
 * Heiko Reese <heiko.reese@kit.edu>