Linux-Forensics-Checklist.md 8.24 KB
Newer Older
Heiko Reese's avatar
Heiko Reese committed
1
2
3
4
5
6
7
8
# KIT-CERT's Checklist for Linux Forensics

## Preliminary Considerations

Forensic investigations of computer hardware is usually divided in two phases:
online forensics (analysis of the running system) and offline forensics
(examination of the permanent storage).

Heiko Reese's avatar
   
Heiko Reese committed
9
10
This document's primary focus is the first phase (online forensics). We assume
that the reader has root access to the compromised machine.
Heiko Reese's avatar
Heiko Reese committed
11

heiko.reese's avatar
   
heiko.reese committed
12
## Find a proper place to store your findings
Heiko Reese's avatar
Heiko Reese committed
13
14
15
16

Every action that interacts with the storage subsystem can potentially destroy
evidence (both data and metadata). Mounting external storage changes the
contents of `/etc/mtab` and the timestamps of the containing directory `/etc`.
heiko.reese's avatar
   
heiko.reese committed
17
18
19
20
21
22
Merely looking at the file (`cat /etc/mtab`) changes the access time of `/etc`.

### Pushing data onto the network

You may push your findings directly onto the network, thus preventing/minimizing
changes to the local filesystems. This only works if the compromized machine is
Heiko Reese's avatar
   
Heiko Reese committed
23
still able to make outgoing connections to the destination server .
heiko.reese's avatar
   
heiko.reese committed
24
25

Open a listener on your server:
Heiko Reese's avatar
   
Heiko Reese committed
26
```sh
heiko.reese's avatar
   
heiko.reese committed
27
28
29
30
nc -l 6789 >> logfilename.txt
```

To send the standard output of a command, simply add this
Heiko Reese's avatar
   
Heiko Reese committed
31
```sh
heiko.reese's avatar
   
heiko.reese committed
32
33
34
 | nc -w 2 name_or_ip_of_server 6789
```

Heiko Reese's avatar
   
Heiko Reese committed
35
Encrypt all data in transition to prevent eavesdropping. Simply insert
Heiko Reese's avatar
   
Heiko Reese committed
36
[`openssl`](https://openssl.org/) into the toolchain:
Heiko Reese's avatar
   
Heiko Reese committed
37
```sh
Heiko Reese's avatar
   
Heiko Reese committed
38
39
nc -l 6789 | openssl enc -aes128 -d -k supersecretpw >> log.txt
```
Heiko Reese's avatar
   
Heiko Reese committed
40
```sh
Heiko Reese's avatar
   
Heiko Reese committed
41
42
43
44
45
 | openssl enc -aes128 -e -k supersecretpw | nc -w 2 name_or_ip_of_server 6789
```

Use [cryptcat](http://cryptcat.sourceforge.net) if it's already available on
the target machine.
heiko.reese's avatar
   
heiko.reese committed
46

Heiko Reese's avatar
   
Heiko Reese committed
47
48
49
50
51
52
53
54
55
56
57
To copy files, use `cat`:

```
cat /usr/bin/rootkit_0.1 | nc …
```

Use `dd` to  transfer whole blockdevices:
```
dd if=/dev/sdx23 | nc…
```

Heiko Reese's avatar
   
Heiko Reese committed
58
59
60
61
Using [fuse sshfs](http://fuse.sourceforge.net/sshfs.html) is discouraged for
two reasons. First, it touches lots of files ($HOME/.ssh/*, /etc). And more
importantly: attackers often change the ssh binaries to intercept passwords.

Heiko Reese's avatar
   
Heiko Reese committed
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
### Collecting data on local storage

If you decide to collect your findings locally, please refrain from using
existing storage of the compromised system. There are two viable options:
external storage like USB-sticks or memory-backed filesystem aka `tmpfs`.
Please save a listing of all mounts in all namespaces before mounting anything.

Check all the different mounts:
```
md5sum /proc/mounts /proc/*/mounts | sort | uniq -d -w 32
```

Get creative to solve this chicken-egg-problem! If you have copy/paste on your
console, simply `cat`the files and copy the aferwards. Don't use screen/tmux,
the touch lots of files. Check for empty pre-existing `tmpfs`-filesystems.

Find a proper location for the mountpoint and Mount your device:
```
mount -t tmpfs none /mnt
# or
mount /dev/sdx1 /mnt
```

Heiko Reese's avatar
   
Heiko Reese committed
85
86
## Collecting evidence

Heiko Reese's avatar
   
Heiko Reese committed
87
88
Collect evidence by saving potentielly interesting parts of the system state.
Start with the most volatile and work your way down:
Heiko Reese's avatar
   
Heiko Reese committed
89
90
91
92

1. network and connection state
1. process state
1. users
Heiko Reese's avatar
   
Heiko Reese committed
93
1. system state and configuration
Heiko Reese's avatar
   
Heiko Reese committed
94
95
96
97
98
99

The following commands assume that you are writing your findings to a local
storage and that your current working directory is set accordingly.

Some programs have rather unstable commandline parameters, please adjust
accordingly (if possible, use `--help` instead of the manpage to find out). You
Heiko Reese's avatar
   
Heiko Reese committed
100
101
102
103
104
105
106
can find the long versions (if applicable) as comments above every command.

Some modern Linux systems have SELinux enabled. Run `getenforce` to find out if
SELinux is enforcing, permissive, or disabled. If the state is enforcing, we
need to get selinux information when applicable. Most tools provide a switch
`-Z` for that. Such commands are marked with a special comment like
`# SELinux: add "-Z"`.
Heiko Reese's avatar
   
Heiko Reese committed
107
108
109
110
111
112
113
114
115
116

### Network state

Get state of existing connections and open sockets:
```sh
# --verbose --wide --extend --timers --program --numeric (--listening)
netstat -v -W -e -o -p -n     > netstat_vWeopn.txt
netstat -v -W -e -o -p -n -l  > netstat_vWeopnl.txt
# same without --numeric
netstat -v -W -e -o -p        > netstat_vWeop.txt
Heiko Reese's avatar
   
Heiko Reese committed
117
netstat -v -W -e -o -p -l     > netstat_vWeop.txt
Heiko Reese's avatar
   
Heiko Reese committed
118
119
```

Heiko Reese's avatar
   
Heiko Reese committed
120
Redo using `ss` if available:
Heiko Reese's avatar
   
Heiko Reese committed
121
122
123
124
125
126

```sh
# --options --extended --processes --info --numeric (--listening )
ss -o -e -p -i -n    > ss_oepin.txt
ss -o -e -p -i -n -l > ss_oepinl.txt
# same without --numeric
Heiko Reese's avatar
   
Heiko Reese committed
127
128
ss -o -e -p -i       > ss_oepi.txt
ss -o -e -p -i -l    > ss_oepil.txt
Heiko Reese's avatar
   
Heiko Reese committed
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
```

Dump arp cache:
```sh
arp -n > arp_n.txt
ip neigh show > ip_neigh_show.txt
```

Get routing-related stuff:
```sh
for i in link addr route rule neigh ntable tunnel tuntap maddr mroute mrule; do
    ip $i list > ip_${i}_l.txt;
done
```

Capture iptable's state:
```sh
# --verbose --numeric --exact --list --table
for t in filter nat mangle raw; do iptables -v -n -x -L -t > iptables_vnxL_t${t}.txt; done
for table in filter mangle raw; do ip6tables -n -t ${table} -L -v -x > ip6tables_nt_${table}.txt; done
for table in filter nat broute; do ebtables -L --Lmac2 --Lc -t ${table} > ebtables_L_Lmac_Lc_t_${table}.txt; done
```

Heiko Reese's avatar
   
Heiko Reese committed
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
### Process State

Save process table:

```sh
ps auxwwwe > ps_auxwwwe.txt
```

Formatting of certain columns seems to be broken in many versions of `ps`, so
we have to add the :xxxxx-postfixes to enforce wide columns. This is not meant
for human consumption:

```sh
ps wwwe -A -o pid,ppid,sess,tname,tpgid,comm,f,uid,euid,rgid,ruid,gid,egid,fgid,ouid,pgid,sgid,suid,supgid,suser,pidns,unit,label,time,lstart,lsession,seat,machine,ni,wchan,etime,%cpu,%mem,cgroup:65535,args:65535 > ps_dump_e.txt
ps www -A -o pid,ppid,sess,tname,tpgid,comm,f,uid,euid,rgid,ruid,gid,egid,fgid,ouid,pgid,sgid,suid,supgid,suser,pidns,unit,label,time,lstart,lsession,seat,machine,ni,wchan,etime,%cpu,%mem,cgroup:65535,args:65535 > ps_dump.txt
```

Something more human-readable:
```sh
pstree -a -l -p -u    > pstree_alpu.txt
pstree -a -l -p -u -Z > pstree_alpuZ.txt
```

Heiko Reese's avatar
   
Heiko Reese committed
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
`lsof` has the most unstable commandline interface. We're planning to include versions for specific linux distributions in the future…

```sh
lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt
```

```sh
# time pid creator limits
 in t p c l; do ipcs -a -${i} > ipcs_a_${i}.txt;done
```

Add this on systems that use [systemd](http://www.freedesktop.org/wiki/Software/systemd/):
```sh
systemctl status -l > systemctl_status_l.txt
```

### Users

```sh
last > last.txt
lastlog > lastlog.txt
who > who.txt
w > w.txt
```

Add this on systems that use systemd:
```sh
loginctl list-sessions > loginctl_list-sessions.txt
for s in $(loginctl list-sessions --no-legend | awk '{print $1}'); do loginctl show-session ${s} > loginctl_show-session_${s}.txt; done
for u in $(loginctl list-users --no-legend | awk '{print $1}'); do loginctl show-user ${u} > loginctl_show-user_${u}.txt; done
```

Heiko Reese's avatar
   
Heiko Reese committed
207
208
209
210
211
212
213
214
215
216
217
218
### System State and Configuration

```sh
dmesg > dmesg.txt
cat /proc/mounts > proc_mounts.txt
# or use the all-namespace-encompassing version
for p in $(md5sum /proc/mounts /proc/*/mounts | sort | uniq -d -w 32 | awk '{print $2}'); do cat $p > ${p////_}; done
cat /proc/mdstat > proc_mdstat.txt
lspci > lspci.txt
uname -a > uname_a.txt
uptime > uptime.txt
```
Heiko Reese's avatar
   
Heiko Reese committed
219

Heiko Reese's avatar
   
Heiko Reese committed
220
221
222
223
224
225
226
227
228
# Dumping suspicious processess

Have a closer look at the process list. Do this for every suspicious process:

Assign it to the `PID` variable for later usage:
```sh
# insert correct PID here!
export PID=12345
```
Heiko Reese's avatar
   
Heiko Reese committed
229

Heiko Reese's avatar
   
Heiko Reese committed
230
231
232
233
234
Stop the process:
```sh
kill -STOP ${PID}
```

Heiko Reese's avatar
   
Heiko Reese committed
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
TODO: Add cgroups-freezer-variant and discuss it (freezer blocks gdb/gcore).

Preserve original location of executable (plus a broken symlink of file was deleted) and the contents:
```sh
ls -l /proc/${PID}/ > proc_${PID}_ls_l.txt
cat /proc/${PID}/exe > proc_${PID}_exe
```

Create a coredump to preserve the process memory:
```sh
gdb -nh -batch -ex gcore -p ${PID}
```

We have not found a way to dump the cores directly into an unnamed pipe and out
into the net. There's a workaround using a named pipe, but you have to find
a good place to put it. We recommend using a tmpfs (either existing or create
a new one for this). As a bonus, this enables us to compress the coredump using
whatever compression software we have available:

```sh
# change path accordingly
MYFIFO=.gcore.fifo
mkfifo ${MYFIFO}

# either run this command in the background (append &) or in another shell
cat ${MYFIFO} | [gzip|bzip2|xz|lzop] -c > core.${PID}

gdb -nh -batch -ex "gcore ${MYFIFO}" -p ${PID}
```

Heiko Reese's avatar
   
Heiko Reese committed
265
266
267
268
269
270
271




#### Authors:
 * Heiko Reese <heiko.reese@kit.edu>
 * Tobias Dussa <tobias.dussa@kit.edu>