Suricata's file extraction on Debian GNU/Linux

Suricata is a high performance open source IDS/IPS project. I used it a long time ago around 2010 when it was released. I've been playing with Snort recently and then found Suricata has a great feature: File extraction. It'd be helpful to those who want to get malware samples from IDS. Anyway, like old days, I want to test it on my own and see how it works on Debian. First things first, I need to build it and see if it works.

Download a latest version of the small installation ISO image. I need to clarify my testing environment: Debian is running on my virtual machine, which has two NICs are eth0 and eth1. Interface eth0 is running on NAT mode and eth1 is running bridge mode.  Debian don't assign any IP addr to eth1.

Because our Debian is the small installation. So we have to install some dependency packages via simply apt-get:

#apt-get install vim openssh-server ethtool libpcap-dev libnfnetlink-dev libnetfilter-queue-dev libdnet-dev libdumbnet-dev libpcre3-dev libpcre3-dbg bison flex make zlib1g-dev autoconf libtool libnss3-dev libnspr4-dev libjansson4 libjansson-dev libyaml-dev libcap-ng0 libcap-ng-dev libnet1-dev libmagic-dev build-essential

Get the source code of Suricata:
#cd /tmp
#wget wget http://www.openinfosecfoundation.org/download/suricata-2.0.tar.gz
#tar zxvf suricata-2.0.tar.gz
#cd suricata-2.0

Compile and installation:
#./configure --enable-nfqueue --enable-gccprotect --prefix=/usr/local/suricata --localstatedir=/var
#make -j3
#make make-full

Edit suricata.yaml:
1, Set the request/response body a litte bigger:
           request-body-limit: 1gb #3072
           response-body-limit: 1gb #3072

2, Enable file extraction:
  - file-store:
      enabled: yes       # set to yes to enable
      log-dir: files    # directory to store the files
      force-magic: no   # force logging magic on all stored files
      force-md5: no     # force logging of md5 checksums
      waldo: file.waldo # waldo file to store the file_id across runs

  # output module to log files tracked in a easily parsable json format
  - file-log:
      enabled: yes
      filename: files-json.log
      append: yes
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

3, Add our "test" rule file( test.rules in this case) into the section "default-rule-path:", like:
default-rule-path: /usr/local/suricata/etc/suricata/rules
rule-files:
 - test.rules
 - botcc.rules

Create a rule file:
/usr/local/suricata/etc/suricata/rules/test.rules

Add one line into test.rules( to save any jpg files) :
alert http any any -> any any (msg:"FILESTORE jpg"; fileext:"jpg"; filestore; sid:6; rev:1;)

Enable the eth1:
#ifconfig eth1 up

According to the Suricata's wiki, we should turn off the TCP GSO:
ethtool -K eth1 tso off
ethtool -K eth1 gro off
ethtool -K eth1 lro off
ethtool -K eth1 gso off
ethtool -K eth1 rx off
ethtool -K eth1 tx off
ethtool -K eth1 sg off
ethtool -K eth1 rxvlan off
ethtool -K eth1 txvlan off
ethtool -N eth1 rx-flow-hash udp4 sdfn
ethtool -N eth1 rx-flow-hash udp6 sdfn
ethtool -n eth1 rx-flow-hash udp6
ethtool -n eth1 rx-flow-hash udp4
ethtool -C eth1 rx-usecs 1000
ethtool -C eth1 adaptive-rx off

Run the Suricata with this command:
/usr/local/suricata/bin/suricata -c /usr/local/suricata/etc/suricata//suricata.yaml -i eth1

Use your firefox/chrome on your host machine, and visit some website, like this.

You should see some girl pictures in /var/log/suricata/files  ;-)

btw: Thanks to Suricata community brings us this fuc*ing awesome IDS/IPS project. Special thanks to Peter Manev.

SYNPROXY: the great DoS mitigation solution

I don't expect to can see a perfect DoS solution in my lifetime;-) As we known, there are tons of commercial gateway-level boxes can mitigate the DoS attack in some ways. But I prefer the combination of x86+GNU/Linux, like the most old school guys. Why? My answer is simple: It's the fuc*ing cheapest solution we have. SYNPROXY is one of  new features of linux kernel 3.13. It's based on netfilter framework and connection tracking. If I understand correctly, SYPROXY should mark the initial SYN packet as UNTRACKED and redirecting them into iptables's action "SYNPROXY"( like ACCEPT, DROP, NF_QUEUE, etc). SYNPROXY would be acting like a network gateway device( router?) to performing the regular TCP x-way handshakes. The original packet will be passing into the dst when handshake process is finished. The contributor Jesper Dangaard Brouer gave us a free speech at DEVCON last month. According to his slide's test result, these numbers are really looking good. I did a little test with my colleague today.

Platform: Debian, SLES-12-beta2
Hardware: Laptop, Server, 100Mbps Switch
Tools: hping3, metasploit

root@d6-test:/home/shawn# iptables -t raw -A PREROUTING -i eth0 -p tcp --dport 8888 --syn -j NOTRACK
root@d6-test:/home/shawn# iptables -A INPUT -i eth0 -p tcp --dport 8888 -m state --state UNTRACKED,INVALID -j SYNPROXY --sack-perm --timestamp --mss 1480 --wscale 7 --ecn
echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose

Result:
Without SYNPROXY: ksoftirq is around 8%-9%
With SYNPROXY: ksoftirq is less than 3%

btw: This result may not be very accurate. Anyway, SYNPROXY works.