Linux Server Security Checklist

Software Engineering

My Setup

Public <-> VPS[Nginx] <->[Wireguard] [Router<->Home-Server]

• Block all ports except the specified in both VPS and Home-Server (NFTables). (DONE)
• Enable SYN Flood protection in both VPS and Home-Server. (DONE)
• Modify kernel settings to make server more secure and performant. (NOTDONE)
• Enable firewaall level rate limiting in VPS. (NFTables) (DONE)
• Implement CrowdSec with crowdsec-firewall-bouncer-nftables in the VPS. (DONE)
• Activate basic DoS protection settings in the router. (DONE)
• Implement application layer rate limits inside Home-Server. (NOTDONE)

• I SHOULD CONSIDER ADDING NGINX TO THE SETUP LIKE THIS GUY: https://www.linkedin.com/pulse/how-why-i-use-crowdsec-protect-my-homelab-%C3%A1kos-luk%C3%A1cs-uhdxf/

• https://doc.crowdsec.net/u/getting_started/installation/linux
• https://gtello.github.io/posts/exposing-server-behind-cgnat/ -> I wont be able to block DoS attacks on the VPS using HTTP requests if I do this.
  • I am at the https://gtello.github.io/posts/exposing-server-behind-cgnat/#configure-wireguard
• https://docs.upzilla.co/posts/guides/2024/04/basic-iptables-configuration-for-ddos-mitigation/
• https://sysopstechnix.com/protect-web-servers-from-ddos-attacks-using-fail2ban/

Update && Upgrade The Server

For Debian:

sudo apt update -y && sudo apt upgrade -y

Disable Root User

https://www.tecmint.com/disable-root-login-in-linux/

https://www.cyberciti.biz/tips/linux-security.html

Monitor The System And Traffic

Only Allow Specific Ports For Incoming Requests

UFW

# Block all incoming traffic using UFW
sudo ufw default deny incoming

# Allow all incoming traffic.
sudo ufw default allow outgoing

# Allow just specific ports for incoming traffic
sudo ufw allow 22/tcp # SSH (MUST)
sudo ufw allow 80/tcp # HTTP
sudo ufw allow 443/tcp # HTTPS
# sudo ufw allow 55107 # Custom Wireguard port

# Enable UFW Firewall
sudo ufw enable

# Verify Firewall Rules
sudo ufw status verbose

NFTables (For VPS)

To use NGINX in the VPS with stream.

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    # Sets for both IPv4 and IPv6 rate limiting
    set ssh_flood_v4 { type ipv4_addr; flags dynamic; timeout 10m; }
    set ssh_flood_v6 { type ipv6_addr; flags dynamic; timeout 10m; }

    set mail_flood_v4 { type ipv4_addr; flags dynamic; timeout 10m; }
    set mail_flood_v6 { type ipv6_addr; flags dynamic; timeout 10m; }

    set web_flood_v4 { type ipv4_addr; flags dynamic; timeout 1m; }
    set web_flood_v6 { type ipv6_addr; flags dynamic; timeout 1m; }

    # 1. Handle Incoming Traffic (Input)
    chain input {
        type filter hook input priority 0; policy drop;

        # Allow established/related traffic (essential for internet)
        ct state { established, related } accept
	# Drop invalid packets
	ct state invalid drop
        
        # Allow all incoming traffic from loopback & wireguard interfaces
        iifname "lo" accept
        iifname "wg0" accept
	# Allow Wireguard Port
	udp dport 55107 accept

	# ICMP (Ping) - Keep ND unlimited for IPv6 stability
        icmp type echo-request limit rate 5/second accept
        icmpv6 type { echo-request } limit rate 5/second accept
        icmpv6 type { nd-neighbor-solicit, nd-neighbor-advert, nd-router-advert } accept

	# SSH
	## Limit max concurrent connections.
        tcp dport { 22, 53752 } meter ssh_connlimit_v4 { ip saddr ct count over 5 } reject with tcp reset
        tcp dport { 22, 53752 } meter ssh_connlimit_v6 { ip6 saddr ct count over 5 } reject with tcp reset
	## Rate limit new connections. One connection permits 3 login attempts.
	ct state new tcp dport { 22, 53752 } update @ssh_flood_v4 { ip saddr limit rate 3/minute burst 3 packets } accept
	ct state new tcp dport { 22, 53752 } update @ssh_flood_v6 { ip6 saddr limit rate 3/minute burst 3 packets } accept

	# Email SMTP (25,465) and IMAP (993)
	## Limit max concurrent connections
        tcp dport { 25, 465, 993 } meter mail_connlimit_v4 { ip saddr ct count over 10 } reject with tcp reset
        tcp dport { 25, 465, 993 } meter mail_connlimit_v6 { ip6 saddr ct count over 10 } reject with tcp reset
	## Rate limit new connections.
	ct state new tcp dport { 25, 465, 993 } update @mail_flood_v4 { ip saddr limit rate 20/minute burst 10 packets } accept
	ct state new tcp dport { 25, 465, 993 } update @mail_flood_v6 { ip6 saddr limit rate 20/minute burst 10 packets } accept

	# HTTP AND HTTPS PORTS
	## Limit max concurrent connections
        tcp dport { 80, 443 } meter web_connlimit_v4 { ip saddr ct count over 50 } reject with tcp reset
        tcp dport { 80, 443 } meter web_connlimit_v6 { ip6 saddr ct count over 50 } reject with tcp reset
	## Rate limit new connections.
	ct state new tcp dport { 80, 443 } update @web_flood_v4 { ip saddr limit rate 50/second burst 30 packets } accept
	ct state new tcp dport { 80, 443 } update @web_flood_v6 { ip6 saddr limit rate 50/second burst 30 packets } accept
    }

    # 2. Handle Forwarding (Routing)
    chain forward {
        type filter hook forward priority 0; policy drop;

	# Allow traffic from Wireguard to Internet
        iifname "wg0" oifname "ens192" accept
        # Allow return traffic
        iifname "ens192" oifname "wg0" ct state established, related accept
    }

    # 3. Handle Outgoing Traffic (Output)
    chain output {
        type filter hook output priority 0; policy accept;   
    }

    # 4. NAT / Masquerade
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        
        # Replace 'ens192' with your actual public interface if different
        ip saddr 192.168.4.0/24 oifname "ens192" masquerade
    }
}

To Forward all traffic to ports directly without Nginx

#!/usr/sbin/nft -f

flush ruleset

table ip nat {
    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;

        # Forward Email and Web traffic to the Go/Mail server over WireGuard
        # This keeps the original Source IP intact
        iifname "ens192" tcp dport { 25, 80, 443, 465, 993 } dnat to 192.168.4.2
    }

    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;

        # Standard masquerade for WG clients going out to internet
        ip saddr 192.168.4.0/24 oifname "ens192" masquerade

        # IMPORTANT: Do NOT masquerade traffic destined for 192.168.4.2
        # so the backend sees the real client IP.
    }
}

table inet filter {
    # Sets for both IPv4 and IPv6 rate limiting
    set ssh_flood_v4 { type ipv4_addr; flags dynamic; timeout 10m; }
    set ssh_flood_v6 { type ipv6_addr; flags dynamic; timeout 10m; }

    set mail_flood_v4 { type ipv4_addr; flags dynamic; timeout 10m; }
    set mail_flood_v6 { type ipv6_addr; flags dynamic; timeout 10m; }

    set web_flood_v4 { type ipv4_addr; flags dynamic; timeout 1m; }
    set web_flood_v6 { type ipv6_addr; flags dynamic; timeout 1m; }

    # 1. Handle Incoming Traffic (Input)
    chain input {
        type filter hook input priority 0; policy drop;

        # Allow established/related traffic (essential for internet)
        ct state { established, related } accept
        # Drop invalid packets
        ct state invalid drop

        # Allow all incoming traffic from loopback & wireguard interfaces
        iifname "lo" accept
        iifname "wg0" accept
        # Allow Wireguard Port
        udp dport 55107 accept

        # ICMP (Ping) - Keep ND unlimited for IPv6 stability
        icmp type echo-request limit rate 5/second accept
        icmpv6 type { echo-request } limit rate 5/second accept
        icmpv6 type { nd-neighbor-solicit, nd-neighbor-advert, nd-router-advert } accept

        # SSH
        ## Limit max concurrent connections.
        tcp dport { 22, 53752 } meter ssh_connlimit_v4 { ip saddr ct count over 5 } reject with tcp reset
        tcp dport { 22, 53752 } meter ssh_connlimit_v6 { ip6 saddr ct count over 5 } reject with tcp reset
        ## Rate limit new connections. One connection permits 3 login attempts.
        ct state new tcp dport { 22, 53752 } update @ssh_flood_v4 { ip saddr limit rate 3/minute burst 3 packets } accept
        ct state new tcp dport { 22, 53752 } update @ssh_flood_v6 { ip6 saddr limit rate 3/minute burst 3 packets } accept
    }

    # 2. Handle Forwarding (Routing)
    chain forward {
        type filter hook forward priority 0; policy drop;

        # Allow return traffic and established connections
        ct state { established, related } accept

        # Allow traffic from Wireguard to Internet
        iifname "wg0" oifname "ens192" accept

        # Allow incoming Web/Email to the backend server (192.168.4.2)
        # We apply filtering/limits here since it's being forwarded

        # --- Rate Limiting for Forwarded Web Traffic ---
        iifname "ens192" ip daddr 192.168.4.2 tcp dport { 80, 443 } meter web_connlimit_v4 { ip saddr ct count over 50 } reject with tcp reset
        iifname "ens192" ip daddr 192.168.4.2 ct state new tcp dport { 80, 443 } update @web_flood_v4 { ip saddr limit rate 50/second burst 30 packets } accept

        # --- Rate Limiting for Forwarded Mail Traffic ---
        iifname "ens192" ip daddr 192.168.4.2 tcp dport { 25, 465, 993 } meter mail_connlimit_v4 { ip saddr ct count over 10 } reject with tcp reset
        iifname "ens192" ip daddr 192.168.4.2 ct state new tcp dport { 25, 465, 993 } update @mail_flood_v4 { ip saddr limit rate 20/minute burst 10 packets } accept

    }

    # 3. Handle Outgoing Traffic (Output)
    chain output {
        type filter hook output priority 0; policy accept;
    }
}

Configure Firewall Against DoS Attacks

SYN cookies prevent an attacker from exhausting your server's memory by forcing the state to be saved on the client side. This is your primary defense.

• cat /proc/sys/net/ipv4/tcp_syncookies

TODO: USE IPTABLES! DO NOT USE UFW. OR USE UFW BEFORE AND AFTER RULES!

TODO: LOOK AT THE BASH SCRIPT INSIDE VPS. IT CONTAINS VERY USEFUL INFO.

Setup CrowdSec

https://docs.crowdsec.net/u/getting_started/installation/linux

It will prevent SSH brute-force attacks by blocking the attacker IP. It will also provide an another layer of defense against DoS attacks.

# Install the repositories for the latest crowdsec versions.
curl -s https://install.crowdsec.net | sudo sh

# Install Crowdsec
sudo apt install crowdsec

# Install Remediation Component
sudo apt install crowdsec-firewall-bouncer-iptables

Enable NGINX Acquisitions if you use NGINX:

sudo cscli collections install crowdsecurity/nginx

mkdir -p /etc/crowdsec/acquis.d/
echo "source: file
filenames:
  - /var/log/nginx/access.log
  - /var/log/nginx/error.log
labels:
  type: nginx" > /etc/crowdsec/acquis.d/nginx.yaml

systemctl restart crowdsec.service

Regularly Backup Important Data