Linux-Forensics-Checklist.md 9.12 KB
Newer Older
Heiko Reese's avatar
Heiko Reese committed
1
2
3
4
5
6
7
8
# KIT-CERT's Checklist for Linux Forensics

## Preliminary Considerations

Forensic investigations of computer hardware is usually divided in two phases:
online forensics (analysis of the running system) and offline forensics
(examination of the permanent storage).

Heiko Reese's avatar
   
Heiko Reese committed
9
10
This document's primary focus is the first phase (online forensics). We assume
that the reader has root access to the compromised machine.
Heiko Reese's avatar
Heiko Reese committed
11

heiko.reese's avatar
   
heiko.reese committed
12
## Find a proper place to store your findings
Heiko Reese's avatar
Heiko Reese committed
13
14
15
16

Every action that interacts with the storage subsystem can potentially destroy
evidence (both data and metadata). Mounting external storage changes the
contents of `/etc/mtab` and the timestamps of the containing directory `/etc`.
heiko.reese's avatar
   
heiko.reese committed
17
18
19
20
21
22
Merely looking at the file (`cat /etc/mtab`) changes the access time of `/etc`.

### Pushing data onto the network

You may push your findings directly onto the network, thus preventing/minimizing
changes to the local filesystems. This only works if the compromized machine is
Heiko Reese's avatar
   
Heiko Reese committed
23
still able to make outgoing connections to the destination server .
heiko.reese's avatar
   
heiko.reese committed
24
25

Open a listener on your server:
Heiko Reese's avatar
   
Heiko Reese committed
26
```sh
heiko.reese's avatar
   
heiko.reese committed
27
28
29
30
nc -l 6789 >> logfilename.txt
```

To send the standard output of a command, simply add this
Heiko Reese's avatar
   
Heiko Reese committed
31
```sh
heiko.reese's avatar
   
heiko.reese committed
32
33
34
 | nc -w 2 name_or_ip_of_server 6789
```

Heiko Reese's avatar
   
Heiko Reese committed
35
Encrypt all data in transition to prevent eavesdropping. Simply insert
Heiko Reese's avatar
   
Heiko Reese committed
36
[`openssl`](https://openssl.org/) into the toolchain:
Heiko Reese's avatar
   
Heiko Reese committed
37
```sh
Heiko Reese's avatar
   
Heiko Reese committed
38
39
nc -l 6789 | openssl enc -aes128 -d -k supersecretpw >> log.txt
```
Heiko Reese's avatar
   
Heiko Reese committed
40
```sh
Heiko Reese's avatar
   
Heiko Reese committed
41
42
43
 | openssl enc -aes128 -e -k supersecretpw | nc -w 2 name_or_ip_of_server 6789
```

Heiko Reese's avatar
   
Heiko Reese committed
44
45
Use [cryptcat](http://cryptcat.sourceforge.net) if it's available on the
compromised machine.
heiko.reese's avatar
   
heiko.reese committed
46

Heiko Reese's avatar
   
Heiko Reese committed
47
48
49
50
51
52
53
54
55
56
57
To copy files, use `cat`:

```
cat /usr/bin/rootkit_0.1 | nc …
```

Use `dd` to  transfer whole blockdevices:
```
dd if=/dev/sdx23 | nc…
```

Heiko Reese's avatar
   
Heiko Reese committed
58
59
60
61
Using [fuse sshfs](http://fuse.sourceforge.net/sshfs.html) is discouraged for
two reasons. First, it touches lots of files ($HOME/.ssh/*, /etc). And more
importantly: attackers often change the ssh binaries to intercept passwords.

Heiko Reese's avatar
   
Heiko Reese committed
62
63
64
The same problems apply to the `… | ssh user@host 'cat > /my/destination/file`
approach.

Heiko Reese's avatar
   
Heiko Reese committed
65
66
67
68
69
70
71
72
73
74
75
76
77
78
### Collecting data on local storage

If you decide to collect your findings locally, please refrain from using
existing storage of the compromised system. There are two viable options:
external storage like USB-sticks or memory-backed filesystem aka `tmpfs`.
Please save a listing of all mounts in all namespaces before mounting anything.

Check all the different mounts:
```
md5sum /proc/mounts /proc/*/mounts | sort | uniq -d -w 32
```

Get creative to solve this chicken-egg-problem! If you have copy/paste on your
console, simply `cat`the files and copy the aferwards. Don't use screen/tmux,
Heiko Reese's avatar
typo    
Heiko Reese committed
79
they touch lots of files. Check for empty pre-existing `tmpfs`-filesystems.
Heiko Reese's avatar
   
Heiko Reese committed
80
81
82
83
84
85
86
87

Find a proper location for the mountpoint and Mount your device:
```
mount -t tmpfs none /mnt
# or
mount /dev/sdx1 /mnt
```

Heiko Reese's avatar
   
Heiko Reese committed
88
89
## Collecting evidence

Heiko Reese's avatar
   
Heiko Reese committed
90
91
Collect evidence by saving potentielly interesting parts of the system state.
Start with the most volatile and work your way down:
Heiko Reese's avatar
   
Heiko Reese committed
92
93
94
95

1. network and connection state
1. process state
1. users
Heiko Reese's avatar
   
Heiko Reese committed
96
1. system state and configuration
Heiko Reese's avatar
   
Heiko Reese committed
97
98
99
100
101
102

The following commands assume that you are writing your findings to a local
storage and that your current working directory is set accordingly.

Some programs have rather unstable commandline parameters, please adjust
accordingly (if possible, use `--help` instead of the manpage to find out). You
Heiko Reese's avatar
   
Heiko Reese committed
103
104
105
106
107
108
109
can find the long versions (if applicable) as comments above every command.

Some modern Linux systems have SELinux enabled. Run `getenforce` to find out if
SELinux is enforcing, permissive, or disabled. If the state is enforcing, we
need to get selinux information when applicable. Most tools provide a switch
`-Z` for that. Such commands are marked with a special comment like
`# SELinux: add "-Z"`.
Heiko Reese's avatar
   
Heiko Reese committed
110
111
112
113
114
115
116
117
118
119

### Network state

Get state of existing connections and open sockets:
```sh
# --verbose --wide --extend --timers --program --numeric (--listening)
netstat -v -W -e -o -p -n     > netstat_vWeopn.txt
netstat -v -W -e -o -p -n -l  > netstat_vWeopnl.txt
# same without --numeric
netstat -v -W -e -o -p        > netstat_vWeop.txt
Heiko Reese's avatar
   
Heiko Reese committed
120
netstat -v -W -e -o -p -l     > netstat_vWeop.txt
Heiko Reese's avatar
   
Heiko Reese committed
121
122
```

Heiko Reese's avatar
   
Heiko Reese committed
123
Redo using `ss` if available:
Heiko Reese's avatar
   
Heiko Reese committed
124
125
126
127
128
129

```sh
# --options --extended --processes --info --numeric (--listening )
ss -o -e -p -i -n    > ss_oepin.txt
ss -o -e -p -i -n -l > ss_oepinl.txt
# same without --numeric
Heiko Reese's avatar
   
Heiko Reese committed
130
131
ss -o -e -p -i       > ss_oepi.txt
ss -o -e -p -i -l    > ss_oepil.txt
Heiko Reese's avatar
   
Heiko Reese committed
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
```

Dump arp cache:
```sh
arp -n > arp_n.txt
ip neigh show > ip_neigh_show.txt
```

Get routing-related stuff:
```sh
for i in link addr route rule neigh ntable tunnel tuntap maddr mroute mrule; do
    ip $i list > ip_${i}_l.txt;
done
```

Capture iptable's state:
```sh
# --verbose --numeric --exact --list --table
for t in filter nat mangle raw; do iptables -v -n -x -L -t > iptables_vnxL_t${t}.txt; done
for table in filter mangle raw; do ip6tables -n -t ${table} -L -v -x > ip6tables_nt_${table}.txt; done
for table in filter nat broute; do ebtables -L --Lmac2 --Lc -t ${table} > ebtables_L_Lmac_Lc_t_${table}.txt; done
```

Heiko Reese's avatar
   
Heiko Reese committed
155
156
157
158
159
160
161
162
163
164
165
166
167
### Process State

Save process table:

```sh
ps auxwwwe > ps_auxwwwe.txt
```

Formatting of certain columns seems to be broken in many versions of `ps`, so
we have to add the :xxxxx-postfixes to enforce wide columns. This is not meant
for human consumption:

```sh
168
169
170
171
172
173
ps wwwe -A -o pid,ppid,sess,tname,tpgid,comm,f,uid,euid,rgid,ruid,gid,egid,fgid,\
ouid,pgid,sgid,suid,supgid,suser,pidns,unit,label,time,lstart,lsession,seat,\
machine,ni,wchan,etime,%cpu,%mem,cgroup:65535,args:65535 > ps_dump_e.txt
ps www -A -o pid,ppid,sess,tname,tpgid,comm,f,uid,euid,rgid,ruid,gid,egid,fgid,\
ouid,pgid,sgid,suid,supgid,suser,pidns,unit,label,time,lstart,lsession,seat,\
machine,ni,wchan,etime,%cpu,%mem,cgroup:65535,args:65535 > ps_dump.txt
Heiko Reese's avatar
   
Heiko Reese committed
174
175
176
177
178
179
180
181
```

Something more human-readable:
```sh
pstree -a -l -p -u    > pstree_alpu.txt
pstree -a -l -p -u -Z > pstree_alpuZ.txt
```

Heiko Reese's avatar
   
Heiko Reese committed
182
183
184
185
186
187
188
189
`lsof` has the most unstable commandline interface. We're planning to include versions for specific linux distributions in the future…

```sh
lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt
```

```sh
# time pid creator limits
Heiko Reese's avatar
typo    
Heiko Reese committed
190
for i in t p c l; do ipcs -a -${i} > ipcs_a_${i}.txt;done
Heiko Reese's avatar
   
Heiko Reese committed
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
```

Add this on systems that use [systemd](http://www.freedesktop.org/wiki/Software/systemd/):
```sh
systemctl status -l > systemctl_status_l.txt
```

### Users

```sh
last > last.txt
lastlog > lastlog.txt
who > who.txt
w > w.txt
```

Add this on systems that use systemd:
```sh
loginctl list-sessions > loginctl_list-sessions.txt
210
211
212
213
214
215
for s in $(loginctl list-sessions --no-legend | awk '{print $1}'); do
    loginctl show-session ${s} > loginctl_show-session_${s}.txt;
done
for u in $(loginctl list-users --no-legend | awk '{print $1}'); do
    loginctl show-user ${u} > loginctl_show-user_${u}.txt;
done
Heiko Reese's avatar
   
Heiko Reese committed
216
217
```

Heiko Reese's avatar
   
Heiko Reese committed
218
219
220
221
222
223
### System State and Configuration

```sh
dmesg > dmesg.txt
cat /proc/mounts > proc_mounts.txt
# or use the all-namespace-encompassing version
224
225
226
for p in $(md5sum /proc/mounts /proc/*/mounts | sort | uniq -d -w 32 | awk '{print $2}'); do
    cat $p > ${p////_};
done
Heiko Reese's avatar
   
Heiko Reese committed
227
228
229
230
231
cat /proc/mdstat > proc_mdstat.txt
lspci > lspci.txt
uname -a > uname_a.txt
uptime > uptime.txt
```
Heiko Reese's avatar
   
Heiko Reese committed
232

Heiko Reese's avatar
   
Heiko Reese committed
233
234
# Dumping suspicious processess

Heiko Reese's avatar
   
Heiko Reese committed
235
236
Have a closer look at the process list. Do this for every suspicious process
(assign pid to the `PID` variable beforehand):
Heiko Reese's avatar
   
Heiko Reese committed
237
238
239
240
```sh
# insert correct PID here!
export PID=12345
```
Heiko Reese's avatar
   
Heiko Reese committed
241

Heiko Reese's avatar
   
Heiko Reese committed
242
243
244
245
246
Stop the process:
```sh
kill -STOP ${PID}
```

Heiko Reese's avatar
   
Heiko Reese committed
247
248
249
250
251
252
253
254
255
256
257
258
259
260
TODO: Add cgroups-freezer-variant and discuss it (freezer blocks gdb/gcore).

Preserve original location of executable (plus a broken symlink of file was deleted) and the contents:
```sh
ls -l /proc/${PID}/ > proc_${PID}_ls_l.txt
cat /proc/${PID}/exe > proc_${PID}_exe
```

Create a coredump to preserve the process memory:
```sh
gdb -nh -batch -ex gcore -p ${PID}
```

We have not found a way to dump the cores directly into an unnamed pipe and out
261
262
into the net. Using FIFOs does not work because gdb needs to seek within the
file while writing it. Using a tmpfs might fails because some coredumps can get
Heiko Reese's avatar
   
Heiko Reese committed
263
264
265
266
267
268
269
270
271
272
273
274
275
pretty big. There are no widely available and stable compressing filesystems
available for linux at the time of writing.

Check for shared memory segments:
```sh
# look for /dev/shms
less /proc/${PID}/map
```

Save some more state information about the process. The available data in the
`/proc/$PID/` of the procfs varies between different kernel versions. Please
check `/proc/self` to what's available and adjust the next commandline
accordingly.
Heiko Reese's avatar
   
Heiko Reese committed
276

Heiko Reese's avatar
   
Heiko Reese committed
277
278
279
280
281
282
283
284
285
286
287
```sh
# use "tar -cf - /proc/… | …" to pipe the tarball to stdout
tar cf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,\
loginuid,maps,mountinfo,sched,schedstat,sessionid,smaps,stack,stat,statm,status,\
syscall,wchan}
```

Have a look at the open files. If the file has been deleted, ls will append
` (deleted)` to the destination filename. The contents can still be accessed
using the symlinks in `/proc/${PID}/fd`. This often happens with malware
written in interpreted languages like perl and python. Save all interesting
Heiko Reese's avatar
typo    
Heiko Reese committed
288
289
open files now:
```sh
Heiko Reese's avatar
   
Heiko Reese committed
290
291
292
293
294
ls -l /proc/${PID}/fd > proc_${PID}_fd.txt
# copy interesting open files, substitute MYFD with file descriptor number
MYFD=1234
cat /proc/${PID}/${MYFD}> proc_${PID}_fd_${MYFD}
```
Heiko Reese's avatar
   
Heiko Reese committed
295
296
297
298


#### Authors:
 * Heiko Reese <heiko.reese@kit.edu>