Linux-Forensics-Checklist.md 15.1 KB
Newer Older
Heiko Reese's avatar
Heiko Reese committed
1
# KIT-CERT Linux Forensics Checklist
Heiko Reese's avatar
Heiko Reese committed
2

Heiko Reese's avatar
Heiko Reese committed
3 4 5 6 7 8 9
This document outlines [our](https://www.cert.kit.edu) initial actions to
investigate a potentially compromised Linux system. It assumes that the reader
has a good technical understanding of how Linux systems work. This is basically
a copy/paste list of things to do in a certain order.

Make sure you understand each action point before you reenact it..

Heiko Reese's avatar
Heiko Reese committed
10 11 12 13 14 15
## Preliminary Considerations

Forensic investigations of computer hardware is usually divided in two phases:
online forensics (analysis of the running system) and offline forensics
(examination of the permanent storage).

Heiko Reese's avatar
Heiko Reese committed
16 17
This document's primary focus is the data gathering aspect of the first phase.
We assume that the reader has root access to the compromised machine.
Heiko Reese's avatar
Heiko Reese committed
18

heiko.reese's avatar
heiko.reese committed
19
## Find a proper place to store your findings
Heiko Reese's avatar
Heiko Reese committed
20 21 22 23

Every action that interacts with the storage subsystem can potentially destroy
evidence (both data and metadata). Mounting external storage changes the
contents of `/etc/mtab` and the timestamps of the containing directory `/etc`.
heiko.reese's avatar
heiko.reese committed
24 25 26 27 28 29
Merely looking at the file (`cat /etc/mtab`) changes the access time of `/etc`.

### Pushing data onto the network

You may push your findings directly onto the network, thus preventing/minimizing
changes to the local filesystems. This only works if the compromized machine is
Heiko Reese's avatar
Heiko Reese committed
30
still able to make outgoing connections to the destination server .
heiko.reese's avatar
heiko.reese committed
31 32

Open a listener on your server:
Heiko Reese's avatar
Heiko Reese committed
33
```sh
heiko.reese's avatar
heiko.reese committed
34 35 36 37
nc -l 6789 >> logfilename.txt
```

To send the standard output of a command, simply add this
Heiko Reese's avatar
Heiko Reese committed
38
```sh
heiko.reese's avatar
heiko.reese committed
39 40 41
 | nc -w 2 name_or_ip_of_server 6789
```

Heiko Reese's avatar
Heiko Reese committed
42
Encrypt all data in transition to prevent eavesdropping. Simply insert
Heiko Reese's avatar
Heiko Reese committed
43
[`openssl`](https://openssl.org/) into the toolchain:
Heiko Reese's avatar
Heiko Reese committed
44
```sh
Heiko Reese's avatar
Heiko Reese committed
45 46
nc -l 6789 | openssl enc -aes128 -d -k supersecretpw >> log.txt
```
Heiko Reese's avatar
Heiko Reese committed
47
```sh
Heiko Reese's avatar
Heiko Reese committed
48 49 50
 | openssl enc -aes128 -e -k supersecretpw | nc -w 2 name_or_ip_of_server 6789
```

Heiko Reese's avatar
Heiko Reese committed
51 52
Use [cryptcat](http://cryptcat.sourceforge.net) if it's available on the
compromised machine.
heiko.reese's avatar
heiko.reese committed
53

Heiko Reese's avatar
Heiko Reese committed
54 55 56 57 58 59 60 61 62 63 64
To copy files, use `cat`:

```
cat /usr/bin/rootkit_0.1 | nc …
```

Use `dd` to  transfer whole blockdevices:
```
dd if=/dev/sdx23 | nc…
```

Heiko Reese's avatar
Heiko Reese committed
65 66 67 68
Using [fuse sshfs](http://fuse.sourceforge.net/sshfs.html) is discouraged for
two reasons. First, it touches lots of files ($HOME/.ssh/*, /etc). And more
importantly: attackers often change the ssh binaries to intercept passwords.

Heiko Reese's avatar
Heiko Reese committed
69 70 71
The same problems apply to the `… | ssh user@host 'cat > /my/destination/file`
approach.

Heiko Reese's avatar
Heiko Reese committed
72 73 74 75 76 77 78 79 80 81 82 83 84 85
### Collecting data on local storage

If you decide to collect your findings locally, please refrain from using
existing storage of the compromised system. There are two viable options:
external storage like USB-sticks or memory-backed filesystem aka `tmpfs`.
Please save a listing of all mounts in all namespaces before mounting anything.

Check all the different mounts:
```
md5sum /proc/mounts /proc/*/mounts | sort | uniq -d -w 32
```

Get creative to solve this chicken-egg-problem! If you have copy/paste on your
console, simply `cat`the files and copy the aferwards. Don't use screen/tmux,
Heiko Reese's avatar
Heiko Reese committed
86
they touch lots of files. Check for empty pre-existing `tmpfs`-filesystems.
Heiko Reese's avatar
Heiko Reese committed
87 88 89 90 91 92 93 94

Find a proper location for the mountpoint and Mount your device:
```
mount -t tmpfs none /mnt
# or
mount /dev/sdx1 /mnt
```

Heiko Reese's avatar
Heiko Reese committed
95 96
## Collecting evidence

Heiko Reese's avatar
Heiko Reese committed
97 98
Collect evidence by saving potentielly interesting parts of the system state.
Start with the most volatile and work your way down:
Heiko Reese's avatar
Heiko Reese committed
99 100 101 102

1. network and connection state
1. process state
1. users
Heiko Reese's avatar
Heiko Reese committed
103
1. system state and configuration
Heiko Reese's avatar
Heiko Reese committed
104 105 106 107 108 109

The following commands assume that you are writing your findings to a local
storage and that your current working directory is set accordingly.

Some programs have rather unstable commandline parameters, please adjust
accordingly (if possible, use `--help` instead of the manpage to find out). You
Heiko Reese's avatar
Heiko Reese committed
110 111 112 113 114
can find the long versions (if applicable) as comments above every command.

Some modern Linux systems have SELinux enabled. Run `getenforce` to find out if
SELinux is enforcing, permissive, or disabled. If the state is enforcing, we
need to get selinux information when applicable. Most tools provide a switch
Heiko Reese's avatar
Heiko Reese committed
115
`-Z` for that.
Heiko Reese's avatar
Heiko Reese committed
116 117 118 119 120 121 122 123 124 125

### Network state

Get state of existing connections and open sockets:
```sh
# --verbose --wide --extend --timers --program --numeric (--listening)
netstat -v -W -e -o -p -n     > netstat_vWeopn.txt
netstat -v -W -e -o -p -n -l  > netstat_vWeopnl.txt
# same without --numeric
netstat -v -W -e -o -p        > netstat_vWeop.txt
126
netstat -v -W -e -o -p -l     > netstat_vWeopl.txt
Heiko Reese's avatar
Heiko Reese committed
127 128
```

Heiko Reese's avatar
Heiko Reese committed
129
Redo using `ss` if available:
Heiko Reese's avatar
Heiko Reese committed
130 131 132 133 134 135

```sh
# --options --extended --processes --info --numeric (--listening )
ss -o -e -p -i -n    > ss_oepin.txt
ss -o -e -p -i -n -l > ss_oepinl.txt
# same without --numeric
Heiko Reese's avatar
Heiko Reese committed
136 137
ss -o -e -p -i       > ss_oepi.txt
ss -o -e -p -i -l    > ss_oepil.txt
Heiko Reese's avatar
Heiko Reese committed
138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155
```

Dump arp cache:
```sh
arp -n > arp_n.txt
ip neigh show > ip_neigh_show.txt
```

Get routing-related stuff:
```sh
for i in link addr route rule neigh ntable tunnel tuntap maddr mroute mrule; do
    ip $i list > ip_${i}_l.txt;
done
```

Capture iptable's state:
```sh
# --verbose --numeric --exact --list --table
156
for table in filter nat mangle raw; do iptables -v -n -x -L -t ${table} > iptables_vnxL_t${table}.txt; done
Heiko Reese's avatar
Heiko Reese committed
157
for table in filter mangle raw; do ip6tables -n -t ${table} -L -v -x > ip6tables_nt_${table}.txt; done
158
for table in filter nat broute; do ebtables -t ${table} -L --Lmac2 --Lc > ebtables_L_Lmac_Lc_t_${table}.txt; done
Heiko Reese's avatar
Heiko Reese committed
159 160
```

heiko.reese's avatar
heiko.reese committed
161 162 163 164 165
Dump ipsets (commonly usef by fail2ban and firewalld):
```sh
ipset list > ipset_list.txt
```

Heiko Reese's avatar
Heiko Reese committed
166 167 168 169 170 171 172 173 174 175 176 177 178
### Process State

Save process table:

```sh
ps auxwwwe > ps_auxwwwe.txt
```

Formatting of certain columns seems to be broken in many versions of `ps`, so
we have to add the :xxxxx-postfixes to enforce wide columns. This is not meant
for human consumption:

```sh
179 180 181 182 183 184
ps wwwe -A -o pid,ppid,sess,tname,tpgid,comm,f,uid,euid,rgid,ruid,gid,egid,fgid,\
ouid,pgid,sgid,suid,supgid,suser,pidns,unit,label,time,lstart,lsession,seat,\
machine,ni,wchan,etime,%cpu,%mem,cgroup:65535,args:65535 > ps_dump_e.txt
ps www -A -o pid,ppid,sess,tname,tpgid,comm,f,uid,euid,rgid,ruid,gid,egid,fgid,\
ouid,pgid,sgid,suid,supgid,suser,pidns,unit,label,time,lstart,lsession,seat,\
machine,ni,wchan,etime,%cpu,%mem,cgroup:65535,args:65535 > ps_dump.txt
Heiko Reese's avatar
Heiko Reese committed
185 186 187 188 189 190 191 192
```

Something more human-readable:
```sh
pstree -a -l -p -u    > pstree_alpu.txt
pstree -a -l -p -u -Z > pstree_alpuZ.txt
```

Heiko Reese's avatar
Heiko Reese committed
193
`lsof` has the most unstable commandline interface. We're planning to include versions for specific Linux distributions in the future…
Heiko Reese's avatar
Heiko Reese committed
194 195 196 197 198 199 200

```sh
lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt
```

```sh
# time pid creator limits
Heiko Reese's avatar
Heiko Reese committed
201
for i in t p c l; do ipcs -a -${i} > ipcs_a_${i}.txt;done
Heiko Reese's avatar
Heiko Reese committed
202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220
```

Add this on systems that use [systemd](http://www.freedesktop.org/wiki/Software/systemd/):
```sh
systemctl status -l > systemctl_status_l.txt
```

### Users

```sh
last > last.txt
lastlog > lastlog.txt
who > who.txt
w > w.txt
```

Add this on systems that use systemd:
```sh
loginctl list-sessions > loginctl_list-sessions.txt
221 222 223 224 225 226
for s in $(loginctl list-sessions --no-legend | awk '{print $1}'); do
    loginctl show-session ${s} > loginctl_show-session_${s}.txt;
done
for u in $(loginctl list-users --no-legend | awk '{print $1}'); do
    loginctl show-user ${u} > loginctl_show-user_${u}.txt;
done
Heiko Reese's avatar
Heiko Reese committed
227 228
```

Heiko Reese's avatar
Heiko Reese committed
229 230 231 232 233 234
### System State and Configuration

```sh
dmesg > dmesg.txt
cat /proc/mounts > proc_mounts.txt
# or use the all-namespace-encompassing version
235 236 237
for p in $(md5sum /proc/mounts /proc/*/mounts | sort | uniq -d -w 32 | awk '{print $2}'); do
    cat $p > ${p////_};
done
Heiko Reese's avatar
Heiko Reese committed
238 239 240 241 242
cat /proc/mdstat > proc_mdstat.txt
lspci > lspci.txt
uname -a > uname_a.txt
uptime > uptime.txt
```
Heiko Reese's avatar
Heiko Reese committed
243

Heiko Reese's avatar
Heiko Reese committed
244
### Dumping suspicious processess
Heiko Reese's avatar
Heiko Reese committed
245

Heiko Reese's avatar
Heiko Reese committed
246 247
Have a closer look at the process list. Do this for every suspicious process
(assign pid to the `PID` variable beforehand):
Heiko Reese's avatar
Heiko Reese committed
248 249 250 251
```sh
# insert correct PID here!
export PID=12345
```
Heiko Reese's avatar
Heiko Reese committed
252

Heiko Reese's avatar
Heiko Reese committed
253 254 255 256 257
Stop the process:
```sh
kill -STOP ${PID}
```

Heiko Reese's avatar
Heiko Reese committed
258 259 260 261 262 263 264 265
Note: While the reception of a  `SIGSTOP` cannot be prevented by the receiving
process, it can be observed from the outside. This might cause a controlling
parent-process to react accordingly. Modern systems provide an alternative method via
[cgroups](https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt),
especially the [freezer
subsystem](https://www.kernel.org/doc/Documentation/cgroups/freezer-subsystem.txt).
There is one caveat: `ptrace()`ing a frozen process will block, rendering
this approach unfit for our purposes. It might nevertheless prove beneficial to
heiko.reese's avatar
heiko.reese committed
266 267 268
freeze everything first and then sort out the mess afterwards. We often see malware
that forks children at a high frequency that are hard to stop inas a whole.
This might be a way to tackle this problem; watch this space for updates :-)
Heiko Reese's avatar
Heiko Reese committed
269 270 271 272 273 274 275 276 277

Preserve original location of executable (plus a broken symlink of file was deleted) and the contents:
```sh
ls -l /proc/${PID}/ > proc_${PID}_ls_l.txt
cat /proc/${PID}/exe > proc_${PID}_exe
```

Create a coredump to preserve the process memory:
```sh
278 279 280
gcore ${PID}
# some systems don't have gcore, use this instead:
gdb -nx -batch -ex "attach ${PID}" -ex "gcore core.${PID}" -ex detach -ex quit
Heiko Reese's avatar
Heiko Reese committed
281 282 283
```

We have not found a way to dump the cores directly into an unnamed pipe and out
284 285
into the net. Using FIFOs does not work because gdb needs to seek within the
file while writing it. Using a tmpfs might fails because some coredumps can get
Heiko Reese's avatar
Heiko Reese committed
286
pretty big. There are no widely available and stable compressing filesystems
Heiko Reese's avatar
Heiko Reese committed
287
available for Linux at the time of writing.
Heiko Reese's avatar
Heiko Reese committed
288 289 290 291

Check for shared memory segments:
```sh
# look for /dev/shms
292
cat /proc/${PID}/maps
Heiko Reese's avatar
Heiko Reese committed
293 294 295 296 297 298
```

Save some more state information about the process. The available data in the
`/proc/$PID/` of the procfs varies between different kernel versions. Please
check `/proc/self` to what's available and adjust the next commandline
accordingly.
Heiko Reese's avatar
Heiko Reese committed
299

Heiko Reese's avatar
Heiko Reese committed
300 301 302 303 304 305 306 307 308 309 310
```sh
# use "tar -cf - /proc/… | …" to pipe the tarball to stdout
tar cf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,\
loginuid,maps,mountinfo,sched,schedstat,sessionid,smaps,stack,stat,statm,status,\
syscall,wchan}
```

Have a look at the open files. If the file has been deleted, ls will append
` (deleted)` to the destination filename. The contents can still be accessed
using the symlinks in `/proc/${PID}/fd`. This often happens with malware
written in interpreted languages like perl and python. Save all interesting
Heiko Reese's avatar
Heiko Reese committed
311 312
open files now:
```sh
Heiko Reese's avatar
Heiko Reese committed
313 314 315 316 317
ls -l /proc/${PID}/fd > proc_${PID}_fd.txt
# copy interesting open files, substitute MYFD with file descriptor number
MYFD=1234
cat /proc/${PID}/${MYFD}> proc_${PID}_fd_${MYFD}
```
Heiko Reese's avatar
Heiko Reese committed
318

Heiko Reese's avatar
Heiko Reese committed
319 320 321
The online forensics parts of this guide end with us pulling power from the
machine. There's usually no need to kill the suspicious processes.

Heiko Reese's avatar
Heiko Reese committed
322
Only kill the process if
Heiko Reese's avatar
Heiko Reese committed
323 324 325 326 327 328 329 330 331 332 333 334 335 336
1. its presence is preventing you from continuing your work (e.g. mountpoints are busy) *and*
1. you are sure that you don't need to access it anymore *or*
1. if you have to keep the machine running after your investigation is finished.



```sh
# are you sure you want to do this?
kill -9 ${PID}
```

### Create a filesystem timeline

If you can't get an image of the system's storage afterwards for offline
Heiko Reese's avatar
Heiko Reese committed
337
forensics, you need to create a rudimentary timeline now. Otherwise, skim over
338
the next parts and process to [dumping LUKS keysp](optionally-retrieve-LUKS-master-keys).
Heiko Reese's avatar
Heiko Reese committed
339 340

Using `find` on
Heiko Reese's avatar
Heiko Reese committed
341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380
a filesystem will `stat()` every file and directory on a filesystem thus
changing all access timestamps. There are (at least) two ways to circumvent
this.

#### Remount all filesystems to readonly

For every filesystem that we want to look at, remount it ro:
```sh
# set mountpoint
MOUNTPOINT=/home
mount -o remount,ro ${MOUNTPOINT}
```

Then create a timeline:
```sh
find "${MOUNTPOINT}" -xdev -print0 | xargs -0 stat -c "%Y %X %Z %A %U %G %n" >> timestamps.dat
```

#### Create readonly aliases

For every filesystem that we want to look at, do this:
```sh
# substitute all mountpoints
ORG_FS=/home
RO_FS=/mnt/ro_fs
mkdir ${RO_FS}
mount --bind ${ORG_FS} ${RO_FS}
mount -o remount,ro ${RO_FS}
find ${RO_FS} -xdev -print0 | xargs -0 stat -c "%Y %X %Z %A %U %G %n" >> timestamps.dat
umount ${RO_FS}
```

#### Cheap alternative: noatime

```sh
mount -o remount,noatime …
```

Use this python program (written by Leif Nixon <TODO: email>) to create a human
readable timeline:
Heiko Reese's avatar
Heiko Reese committed
381
```python
Heiko Reese's avatar
Heiko Reese committed
382
#!/usr/bin/python
Heiko Reese's avatar
Heiko Reese committed
383
# timeline-decorator.py
Heiko Reese's avatar
Heiko Reese committed
384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409

import sys, time

def print_line(flags, t, mode, user, group, name):
    print t, time.ctime(float(t)), flags, mode, user, group, name

for line in sys.stdin:
    line = line[:-1]
    (m, a, c, mode, user, group, name) = line.split(" ", 6)
    if m == a:
        if m == c:
            print_line("mac", m, mode, user, group, name)
        else:
            print_line("ma-", m, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)
    else:
        if m == c:
            print_line("m-c", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
        else:
            print_line("m--", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)

```

Heiko Reese's avatar
Heiko Reese committed
410 411
If you cannot get the system's disks (or images thereof) afterwards, feel free
to look around some more:
Heiko Reese's avatar
Heiko Reese committed
412 413 414 415 416 417 418 419 420 421 422 423

* [chkrootkit]( http://www.chkrootkit.org)
* [OSSEC rootcheck](http://www.ossec.net/main/rootcheck)
* [rkhunter](http://rkhunter.sourceforge.net)
* [Trojanscan](http://www.trojanscan.org)
* [CVE-Checker](http://cvechecker.sourceforge.net/)

If applicable, compare checksums of package management with actual files:

* `debsums` (Debian-based distributions)
* `rpm -Va` (Redhat-based distributions)

Heiko Reese's avatar
Heiko Reese committed
424 425 426 427 428 429 430
### Gather potentially interesting data before shutdown

Copy (or `[dd|tar -cf -] … | …`) things that might be of interest later. There
are no hard rules here, just some general ideas:

* `/etc`
* `/root`
Heiko Reese's avatar
Heiko Reese committed
431
* `/tmp`, `/var/tmp` (generally all public writable directories)
Heiko Reese's avatar
Heiko Reese committed
432 433 434 435 436 437 438
* `/usr/local`
* `/var/log`
* `/home/*/.ssh/authorized_keys`
* sshd binary on disk and in memory (`/proc/$(pidof -s sshd)/exe`)
* anything the tools in the previous step found

TODO: journald, …
Heiko Reese's avatar
Heiko Reese committed
439

440
### Optionally retrieve LUKS master keys
Heiko Reese's avatar
Heiko Reese committed
441

442 443 444 445 446 447 448 449 450
Find out if there are any LUKS-encrypted devices:
```sh
for dev in $(lsblk -n -o KNAME); do
    if cryptsetup isLuks "/dev/${dev}"; then
        echo $dev;
        cryptsetup luksDump /dev/sdc2 > cryptsetup_luksDump_$dev.txt
    fi;
done
```
Heiko Reese's avatar
Heiko Reese committed
451

452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471
There are two options if the system has encrypted disks and the current owner
does not want to share the secret key. Both options requires someone who knows
the encryption keys to enter them once for each device.

Option one: add another keys using
```sh
cryptsetup luksAddKey /dev/…
```

While this changes the LUKS-header on-disk, it does not affect the containing filesystem.

Option two: dump the encryption masterkey. Handle with care!
```sh
for dev in $(lsblk -n -o KNAME); do
    if cryptsetup isLuks "/dev/${dev}"; then
        echo $dev;
        cryptsetup --dump-master-key luksDump "/dev/${dev}" | tee cryptsetup_luksDump_$dev.masterkey
    fi;
done
```
Heiko Reese's avatar
Heiko Reese committed
472

Heiko Reese's avatar
Heiko Reese committed
473 474 475 476 477
## Power down system

*Don't* do a `shutdown` or `poweroff`! Cut the power (hold power button for
several seconds) or »force off« virtual machines.

Heiko Reese's avatar
Heiko Reese committed
478 479 480
## Offline Forensics

TODO:
heiko.reese's avatar
heiko.reese committed
481
* Binaries: strings, hexdump, objdump, elf*, gdb, [rec](http://www.backerstreet.com/rec/rec.htm), [radare2](https://github.com/radare/radare2), IDAPro,…
Heiko Reese's avatar
Heiko Reese committed
482
* Logfiles: grep, sort, log2timeline, …
483
* Autopsy, rkhunter, …
Heiko Reese's avatar
Heiko Reese committed
484 485

#### Authors:
heiko.reese's avatar
heiko.reese committed
486
 * Heiko Reese <heiko.reese@kit.edu>