Linux-Forensics-Checklist.md 15.1 KB
Newer Older
Heiko Reese's avatar
   
Heiko Reese committed
1
# KIT-CERT Linux Forensics Checklist
Heiko Reese's avatar
Heiko Reese committed
2

Heiko Reese's avatar
   
Heiko Reese committed
3
4
5
6
7
8
9
This document outlines [our](https://www.cert.kit.edu) initial actions to
investigate a potentially compromised Linux system. It assumes that the reader
has a good technical understanding of how Linux systems work. This is basically
a copy/paste list of things to do in a certain order.

Make sure you understand each action point before you reenact it..

Heiko Reese's avatar
Heiko Reese committed
10
11
12
13
14
15
## Preliminary Considerations

Forensic investigations of computer hardware is usually divided in two phases:
online forensics (analysis of the running system) and offline forensics
(examination of the permanent storage).

Heiko Reese's avatar
   
Heiko Reese committed
16
17
This document's primary focus is the data gathering aspect of the first phase.
We assume that the reader has root access to the compromised machine.
Heiko Reese's avatar
Heiko Reese committed
18

heiko.reese's avatar
   
heiko.reese committed
19
## Find a proper place to store your findings
Heiko Reese's avatar
Heiko Reese committed
20
21
22
23

Every action that interacts with the storage subsystem can potentially destroy
evidence (both data and metadata). Mounting external storage changes the
contents of `/etc/mtab` and the timestamps of the containing directory `/etc`.
heiko.reese's avatar
   
heiko.reese committed
24
25
26
27
28
29
Merely looking at the file (`cat /etc/mtab`) changes the access time of `/etc`.

### Pushing data onto the network

You may push your findings directly onto the network, thus preventing/minimizing
changes to the local filesystems. This only works if the compromized machine is
Heiko Reese's avatar
   
Heiko Reese committed
30
still able to make outgoing connections to the destination server .
heiko.reese's avatar
   
heiko.reese committed
31
32

Open a listener on your server:
Heiko Reese's avatar
   
Heiko Reese committed
33
```sh
heiko.reese's avatar
   
heiko.reese committed
34
35
36
37
nc -l 6789 >> logfilename.txt
```

To send the standard output of a command, simply add this
Heiko Reese's avatar
   
Heiko Reese committed
38
```sh
heiko.reese's avatar
   
heiko.reese committed
39
40
41
 | nc -w 2 name_or_ip_of_server 6789
```

Heiko Reese's avatar
   
Heiko Reese committed
42
Encrypt all data in transition to prevent eavesdropping. Simply insert
Heiko Reese's avatar
   
Heiko Reese committed
43
[`openssl`](https://openssl.org/) into the toolchain:
Heiko Reese's avatar
   
Heiko Reese committed
44
```sh
Heiko Reese's avatar
   
Heiko Reese committed
45
46
nc -l 6789 | openssl enc -aes128 -d -k supersecretpw >> log.txt
```
Heiko Reese's avatar
   
Heiko Reese committed
47
```sh
Heiko Reese's avatar
   
Heiko Reese committed
48
49
50
 | openssl enc -aes128 -e -k supersecretpw | nc -w 2 name_or_ip_of_server 6789
```

Heiko Reese's avatar
   
Heiko Reese committed
51
52
Use [cryptcat](http://cryptcat.sourceforge.net) if it's available on the
compromised machine.
heiko.reese's avatar
   
heiko.reese committed
53

Heiko Reese's avatar
   
Heiko Reese committed
54
55
56
57
58
59
60
61
62
63
64
To copy files, use `cat`:

```
cat /usr/bin/rootkit_0.1 | nc …
```

Use `dd` to  transfer whole blockdevices:
```
dd if=/dev/sdx23 | nc…
```

Heiko Reese's avatar
   
Heiko Reese committed
65
66
67
68
Using [fuse sshfs](http://fuse.sourceforge.net/sshfs.html) is discouraged for
two reasons. First, it touches lots of files ($HOME/.ssh/*, /etc). And more
importantly: attackers often change the ssh binaries to intercept passwords.

Heiko Reese's avatar
   
Heiko Reese committed
69
70
71
The same problems apply to the `… | ssh user@host 'cat > /my/destination/file`
approach.

Heiko Reese's avatar
   
Heiko Reese committed
72
73
74
75
76
77
78
79
80
81
82
83
84
85
### Collecting data on local storage

If you decide to collect your findings locally, please refrain from using
existing storage of the compromised system. There are two viable options:
external storage like USB-sticks or memory-backed filesystem aka `tmpfs`.
Please save a listing of all mounts in all namespaces before mounting anything.

Check all the different mounts:
```
md5sum /proc/mounts /proc/*/mounts | sort | uniq -d -w 32
```

Get creative to solve this chicken-egg-problem! If you have copy/paste on your
console, simply `cat`the files and copy the aferwards. Don't use screen/tmux,
Heiko Reese's avatar
typo    
Heiko Reese committed
86
they touch lots of files. Check for empty pre-existing `tmpfs`-filesystems.
Heiko Reese's avatar
   
Heiko Reese committed
87
88
89
90
91
92
93
94

Find a proper location for the mountpoint and Mount your device:
```
mount -t tmpfs none /mnt
# or
mount /dev/sdx1 /mnt
```

Heiko Reese's avatar
   
Heiko Reese committed
95
96
## Collecting evidence

Heiko Reese's avatar
   
Heiko Reese committed
97
98
Collect evidence by saving potentielly interesting parts of the system state.
Start with the most volatile and work your way down:
Heiko Reese's avatar
   
Heiko Reese committed
99
100
101
102

1. network and connection state
1. process state
1. users
Heiko Reese's avatar
   
Heiko Reese committed
103
1. system state and configuration
Heiko Reese's avatar
   
Heiko Reese committed
104
105
106
107
108
109

The following commands assume that you are writing your findings to a local
storage and that your current working directory is set accordingly.

Some programs have rather unstable commandline parameters, please adjust
accordingly (if possible, use `--help` instead of the manpage to find out). You
Heiko Reese's avatar
   
Heiko Reese committed
110
111
112
113
114
can find the long versions (if applicable) as comments above every command.

Some modern Linux systems have SELinux enabled. Run `getenforce` to find out if
SELinux is enforcing, permissive, or disabled. If the state is enforcing, we
need to get selinux information when applicable. Most tools provide a switch
Heiko Reese's avatar
   
Heiko Reese committed
115
`-Z` for that.
Heiko Reese's avatar
   
Heiko Reese committed
116
117
118
119
120
121
122
123
124
125

### Network state

Get state of existing connections and open sockets:
```sh
# --verbose --wide --extend --timers --program --numeric (--listening)
netstat -v -W -e -o -p -n     > netstat_vWeopn.txt
netstat -v -W -e -o -p -n -l  > netstat_vWeopnl.txt
# same without --numeric
netstat -v -W -e -o -p        > netstat_vWeop.txt
Heiko Reese's avatar
   
Heiko Reese committed
126
netstat -v -W -e -o -p -l     > netstat_vWeop.txt
Heiko Reese's avatar
   
Heiko Reese committed
127
128
```

Heiko Reese's avatar
   
Heiko Reese committed
129
Redo using `ss` if available:
Heiko Reese's avatar
   
Heiko Reese committed
130
131
132
133
134
135

```sh
# --options --extended --processes --info --numeric (--listening )
ss -o -e -p -i -n    > ss_oepin.txt
ss -o -e -p -i -n -l > ss_oepinl.txt
# same without --numeric
Heiko Reese's avatar
   
Heiko Reese committed
136
137
ss -o -e -p -i       > ss_oepi.txt
ss -o -e -p -i -l    > ss_oepil.txt
Heiko Reese's avatar
   
Heiko Reese committed
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
```

Dump arp cache:
```sh
arp -n > arp_n.txt
ip neigh show > ip_neigh_show.txt
```

Get routing-related stuff:
```sh
for i in link addr route rule neigh ntable tunnel tuntap maddr mroute mrule; do
    ip $i list > ip_${i}_l.txt;
done
```

Capture iptable's state:
```sh
# --verbose --numeric --exact --list --table
156
for table in filter nat mangle raw; do iptables -v -n -x -L -t ${table} > iptables_vnxL_t${table}.txt; done
Heiko Reese's avatar
   
Heiko Reese committed
157
158
159
160
for table in filter mangle raw; do ip6tables -n -t ${table} -L -v -x > ip6tables_nt_${table}.txt; done
for table in filter nat broute; do ebtables -L --Lmac2 --Lc -t ${table} > ebtables_L_Lmac_Lc_t_${table}.txt; done
```

heiko.reese's avatar
heiko.reese committed
161
162
163
164
165
Dump ipsets (commonly usef by fail2ban and firewalld):
```sh
ipset list > ipset_list.txt
```

Heiko Reese's avatar
   
Heiko Reese committed
166
167
168
169
170
171
172
173
174
175
176
177
178
### Process State

Save process table:

```sh
ps auxwwwe > ps_auxwwwe.txt
```

Formatting of certain columns seems to be broken in many versions of `ps`, so
we have to add the :xxxxx-postfixes to enforce wide columns. This is not meant
for human consumption:

```sh
179
180
181
182
183
184
ps wwwe -A -o pid,ppid,sess,tname,tpgid,comm,f,uid,euid,rgid,ruid,gid,egid,fgid,\
ouid,pgid,sgid,suid,supgid,suser,pidns,unit,label,time,lstart,lsession,seat,\
machine,ni,wchan,etime,%cpu,%mem,cgroup:65535,args:65535 > ps_dump_e.txt
ps www -A -o pid,ppid,sess,tname,tpgid,comm,f,uid,euid,rgid,ruid,gid,egid,fgid,\
ouid,pgid,sgid,suid,supgid,suser,pidns,unit,label,time,lstart,lsession,seat,\
machine,ni,wchan,etime,%cpu,%mem,cgroup:65535,args:65535 > ps_dump.txt
Heiko Reese's avatar
   
Heiko Reese committed
185
186
187
188
189
190
191
192
```

Something more human-readable:
```sh
pstree -a -l -p -u    > pstree_alpu.txt
pstree -a -l -p -u -Z > pstree_alpuZ.txt
```

Heiko Reese's avatar
   
Heiko Reese committed
193
`lsof` has the most unstable commandline interface. We're planning to include versions for specific Linux distributions in the future…
Heiko Reese's avatar
   
Heiko Reese committed
194
195
196
197
198
199
200

```sh
lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt
```

```sh
# time pid creator limits
Heiko Reese's avatar
typo    
Heiko Reese committed
201
for i in t p c l; do ipcs -a -${i} > ipcs_a_${i}.txt;done
Heiko Reese's avatar
   
Heiko Reese committed
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
```

Add this on systems that use [systemd](http://www.freedesktop.org/wiki/Software/systemd/):
```sh
systemctl status -l > systemctl_status_l.txt
```

### Users

```sh
last > last.txt
lastlog > lastlog.txt
who > who.txt
w > w.txt
```

Add this on systems that use systemd:
```sh
loginctl list-sessions > loginctl_list-sessions.txt
221
222
223
224
225
226
for s in $(loginctl list-sessions --no-legend | awk '{print $1}'); do
    loginctl show-session ${s} > loginctl_show-session_${s}.txt;
done
for u in $(loginctl list-users --no-legend | awk '{print $1}'); do
    loginctl show-user ${u} > loginctl_show-user_${u}.txt;
done
Heiko Reese's avatar
   
Heiko Reese committed
227
228
```

Heiko Reese's avatar
   
Heiko Reese committed
229
230
231
232
233
234
### System State and Configuration

```sh
dmesg > dmesg.txt
cat /proc/mounts > proc_mounts.txt
# or use the all-namespace-encompassing version
235
236
237
for p in $(md5sum /proc/mounts /proc/*/mounts | sort | uniq -d -w 32 | awk '{print $2}'); do
    cat $p > ${p////_};
done
Heiko Reese's avatar
   
Heiko Reese committed
238
239
240
241
242
cat /proc/mdstat > proc_mdstat.txt
lspci > lspci.txt
uname -a > uname_a.txt
uptime > uptime.txt
```
Heiko Reese's avatar
   
Heiko Reese committed
243

Heiko Reese's avatar
   
Heiko Reese committed
244
### Dumping suspicious processess
Heiko Reese's avatar
   
Heiko Reese committed
245

Heiko Reese's avatar
   
Heiko Reese committed
246
247
Have a closer look at the process list. Do this for every suspicious process
(assign pid to the `PID` variable beforehand):
Heiko Reese's avatar
   
Heiko Reese committed
248
249
250
251
```sh
# insert correct PID here!
export PID=12345
```
Heiko Reese's avatar
   
Heiko Reese committed
252

Heiko Reese's avatar
   
Heiko Reese committed
253
254
255
256
257
Stop the process:
```sh
kill -STOP ${PID}
```

Heiko Reese's avatar
   
Heiko Reese committed
258
259
260
261
262
263
264
265
Note: While the reception of a  `SIGSTOP` cannot be prevented by the receiving
process, it can be observed from the outside. This might cause a controlling
parent-process to react accordingly. Modern systems provide an alternative method via
[cgroups](https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt),
especially the [freezer
subsystem](https://www.kernel.org/doc/Documentation/cgroups/freezer-subsystem.txt).
There is one caveat: `ptrace()`ing a frozen process will block, rendering
this approach unfit for our purposes. It might nevertheless prove beneficial to
heiko.reese's avatar
heiko.reese committed
266
267
268
freeze everything first and then sort out the mess afterwards. We often see malware
that forks children at a high frequency that are hard to stop inas a whole.
This might be a way to tackle this problem; watch this space for updates :-)
Heiko Reese's avatar
   
Heiko Reese committed
269
270
271
272
273
274
275
276
277

Preserve original location of executable (plus a broken symlink of file was deleted) and the contents:
```sh
ls -l /proc/${PID}/ > proc_${PID}_ls_l.txt
cat /proc/${PID}/exe > proc_${PID}_exe
```

Create a coredump to preserve the process memory:
```sh
278
279
280
gcore ${PID}
# some systems don't have gcore, use this instead:
gdb -nx -batch -ex "attach ${PID}" -ex "gcore core.${PID}" -ex detach -ex quit
Heiko Reese's avatar
   
Heiko Reese committed
281
282
283
```

We have not found a way to dump the cores directly into an unnamed pipe and out
284
285
into the net. Using FIFOs does not work because gdb needs to seek within the
file while writing it. Using a tmpfs might fails because some coredumps can get
Heiko Reese's avatar
   
Heiko Reese committed
286
pretty big. There are no widely available and stable compressing filesystems
Heiko Reese's avatar
   
Heiko Reese committed
287
available for Linux at the time of writing.
Heiko Reese's avatar
   
Heiko Reese committed
288
289
290
291

Check for shared memory segments:
```sh
# look for /dev/shms
292
cat /proc/${PID}/maps
Heiko Reese's avatar
   
Heiko Reese committed
293
294
295
296
297
298
```

Save some more state information about the process. The available data in the
`/proc/$PID/` of the procfs varies between different kernel versions. Please
check `/proc/self` to what's available and adjust the next commandline
accordingly.
Heiko Reese's avatar
   
Heiko Reese committed
299

Heiko Reese's avatar
   
Heiko Reese committed
300
301
302
303
304
305
306
307
308
309
310
```sh
# use "tar -cf - /proc/… | …" to pipe the tarball to stdout
tar cf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,\
loginuid,maps,mountinfo,sched,schedstat,sessionid,smaps,stack,stat,statm,status,\
syscall,wchan}
```

Have a look at the open files. If the file has been deleted, ls will append
` (deleted)` to the destination filename. The contents can still be accessed
using the symlinks in `/proc/${PID}/fd`. This often happens with malware
written in interpreted languages like perl and python. Save all interesting
Heiko Reese's avatar
typo    
Heiko Reese committed
311
312
open files now:
```sh
Heiko Reese's avatar
   
Heiko Reese committed
313
314
315
316
317
ls -l /proc/${PID}/fd > proc_${PID}_fd.txt
# copy interesting open files, substitute MYFD with file descriptor number
MYFD=1234
cat /proc/${PID}/${MYFD}> proc_${PID}_fd_${MYFD}
```
Heiko Reese's avatar
   
Heiko Reese committed
318

Heiko Reese's avatar
   
Heiko Reese committed
319
320
321
The online forensics parts of this guide end with us pulling power from the
machine. There's usually no need to kill the suspicious processes.

Heiko Reese's avatar
   
Heiko Reese committed
322
Only kill the process if
Heiko Reese's avatar
   
Heiko Reese committed
323
324
325
326
327
328
329
330
331
332
333
334
335
336
1. its presence is preventing you from continuing your work (e.g. mountpoints are busy) *and*
1. you are sure that you don't need to access it anymore *or*
1. if you have to keep the machine running after your investigation is finished.



```sh
# are you sure you want to do this?
kill -9 ${PID}
```

### Create a filesystem timeline

If you can't get an image of the system's storage afterwards for offline
Heiko Reese's avatar
   
Heiko Reese committed
337
forensics, you need to create a rudimentary timeline now. Otherwise, skim over
338
the next parts and process to [dumping LUKS keysp](optionally-retrieve-LUKS-master-keys).
Heiko Reese's avatar
   
Heiko Reese committed
339
340

Using `find` on
Heiko Reese's avatar
   
Heiko Reese committed
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
a filesystem will `stat()` every file and directory on a filesystem thus
changing all access timestamps. There are (at least) two ways to circumvent
this.

#### Remount all filesystems to readonly

For every filesystem that we want to look at, remount it ro:
```sh
# set mountpoint
MOUNTPOINT=/home
mount -o remount,ro ${MOUNTPOINT}
```

Then create a timeline:
```sh
find "${MOUNTPOINT}" -xdev -print0 | xargs -0 stat -c "%Y %X %Z %A %U %G %n" >> timestamps.dat
```

#### Create readonly aliases

For every filesystem that we want to look at, do this:
```sh
# substitute all mountpoints
ORG_FS=/home
RO_FS=/mnt/ro_fs
mkdir ${RO_FS}
mount --bind ${ORG_FS} ${RO_FS}
mount -o remount,ro ${RO_FS}
find ${RO_FS} -xdev -print0 | xargs -0 stat -c "%Y %X %Z %A %U %G %n" >> timestamps.dat
umount ${RO_FS}
```

#### Cheap alternative: noatime

```sh
mount -o remount,noatime …
```

Use this python program (written by Leif Nixon <TODO: email>) to create a human
readable timeline:
Heiko Reese's avatar
   
Heiko Reese committed
381
```python
Heiko Reese's avatar
   
Heiko Reese committed
382
#!/usr/bin/python
Heiko Reese's avatar
   
Heiko Reese committed
383
# timeline-decorator.py
Heiko Reese's avatar
   
Heiko Reese committed
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409

import sys, time

def print_line(flags, t, mode, user, group, name):
    print t, time.ctime(float(t)), flags, mode, user, group, name

for line in sys.stdin:
    line = line[:-1]
    (m, a, c, mode, user, group, name) = line.split(" ", 6)
    if m == a:
        if m == c:
            print_line("mac", m, mode, user, group, name)
        else:
            print_line("ma-", m, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)
    else:
        if m == c:
            print_line("m-c", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
        else:
            print_line("m--", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)

```

Heiko Reese's avatar
   
Heiko Reese committed
410
411
If you cannot get the system's disks (or images thereof) afterwards, feel free
to look around some more:
Heiko Reese's avatar
   
Heiko Reese committed
412
413
414
415
416
417
418
419
420
421
422
423

* [chkrootkit]( http://www.chkrootkit.org)
* [OSSEC rootcheck](http://www.ossec.net/main/rootcheck)
* [rkhunter](http://rkhunter.sourceforge.net)
* [Trojanscan](http://www.trojanscan.org)
* [CVE-Checker](http://cvechecker.sourceforge.net/)

If applicable, compare checksums of package management with actual files:

* `debsums` (Debian-based distributions)
* `rpm -Va` (Redhat-based distributions)

Heiko Reese's avatar
   
Heiko Reese committed
424
425
426
427
428
429
430
### Gather potentially interesting data before shutdown

Copy (or `[dd|tar -cf -] … | …`) things that might be of interest later. There
are no hard rules here, just some general ideas:

* `/etc`
* `/root`
Heiko Reese's avatar
   
Heiko Reese committed
431
* `/tmp`, `/var/tmp` (generally all public writable directories)
Heiko Reese's avatar
   
Heiko Reese committed
432
433
434
435
436
437
438
* `/usr/local`
* `/var/log`
* `/home/*/.ssh/authorized_keys`
* sshd binary on disk and in memory (`/proc/$(pidof -s sshd)/exe`)
* anything the tools in the previous step found

TODO: journald, …
Heiko Reese's avatar
   
Heiko Reese committed
439

440
### Optionally retrieve LUKS master keys
Heiko Reese's avatar
   
Heiko Reese committed
441

442
443
444
445
446
447
448
449
450
Find out if there are any LUKS-encrypted devices:
```sh
for dev in $(lsblk -n -o KNAME); do
    if cryptsetup isLuks "/dev/${dev}"; then
        echo $dev;
        cryptsetup luksDump /dev/sdc2 > cryptsetup_luksDump_$dev.txt
    fi;
done
```
Heiko Reese's avatar
   
Heiko Reese committed
451

452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
There are two options if the system has encrypted disks and the current owner
does not want to share the secret key. Both options requires someone who knows
the encryption keys to enter them once for each device.

Option one: add another keys using
```sh
cryptsetup luksAddKey /dev/…
```

While this changes the LUKS-header on-disk, it does not affect the containing filesystem.

Option two: dump the encryption masterkey. Handle with care!
```sh
for dev in $(lsblk -n -o KNAME); do
    if cryptsetup isLuks "/dev/${dev}"; then
        echo $dev;
        cryptsetup --dump-master-key luksDump "/dev/${dev}" | tee cryptsetup_luksDump_$dev.masterkey
    fi;
done
```
Heiko Reese's avatar
   
Heiko Reese committed
472

Heiko Reese's avatar
   
Heiko Reese committed
473
474
475
476
477
## Power down system

*Don't* do a `shutdown` or `poweroff`! Cut the power (hold power button for
several seconds) or »force off« virtual machines.

Heiko Reese's avatar
   
Heiko Reese committed
478
479
480
481
482
483
## Offline Forensics

TODO:
* Binaries: strings, hexdump, objdump, elf*, gdb, rec (http://www.backerstreet.com/rec/rec.htm), IDAPro,…
* Logfiles: grep, sort, log2timeline, …
* Autosy, rkhunter, …
Heiko Reese's avatar
   
Heiko Reese committed
484
485

#### Authors:
heiko.reese's avatar
heiko.reese committed
486
 * Heiko Reese <heiko.reese@kit.edu>