Differences

This shows you the differences between two versions of the page.

--- technical:remote_syslog_setup [2025/06/04 18:39] – nugget
+++ technical:remote_syslog_setup [2025/06/11 04:45] (current) – [Going against the Linux Gods] super_stunder
@@ Line 1: / Line 1: @@
-FIXME I have been working on a simple SYSLOG project to gather scanning data off of host around the internet and centralize it.  The no telling what I will do with the data at that point.  Its a project I thought of a while back and I nicknamed it TopTalkers.  Basically this is the setup I am going for, very simple a remote syslog server with two host sending it traffic.
+====== Remote Syslog - IPTables - Tigers and Bears OH MY ======
+I have been working on a simple SYSLOG project to gather scanning data off of host around the internet and centralize it.  The no telling what I will do with the data at that point.  Its a project I thought of a while back and I nicknamed it TopTalkers.  Basically this is the setup I am going for, very simple a remote syslog server with two host sending it traffic.
 <graphviz>
 digraph MyGraph {
-  logger01 -> syslog01
+  syslog01 -> logger01 [dir=back];
-  logger02 -> syslog01
+  syslog01 -> logger02 [dir=back];
+  label = "Basic Concept"
 }
 </graphviz>
@@ Line 16: / Line 20: @@
 ----
+===== Configuring iptables (the hell of logging denies) =====
 <del>So I am going to start this off and load rsyslog and UFW on the logger servers then I will go from there. I am using UFW for basic firewall configs it makes things way easier when building basic rules.
@@ Line 33: / Line 39: @@
 -A OUTPUT -o lo -j ACCEPT
 </cli>
+==== Saving iptables config and setting persistence after reboots ====
 After you have tested access back into your device over SSH you can now issue the persistent command to maintain the rules at startup. Turns on iptables-persistent isn't installed by default.  So install it then run it and accept the warning to save the current policy.
@@ Line 46: / Line 54: @@
 </cli>
+==== Displaying iptables rules ====
 While digging into the rules I was just using the iptables -L command to list the policy.  This command gives some basic down and dirty output and I have found using iptabels -L -v with the verbose option gives much better results with a packet counter and the interface information that is left out of the basic -L command.
-<cli>
+<cli>root@logger02:~# iptables -L -v
-root@logger02:~# iptables -L -v
+Chain INPUT (policy DROP 14136 packets, 706K bytes)
-Chain INPUT (policy DROP 2559 packets, 149K bytes)
  pkts bytes target     prot opt in     out     source               destination
-1340K LOG        all  --  any    any     anywhere             anywhere             LOG level debug prefix "FIREWALL_LOG: "
+4901K ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED
-     0 ACCEPT     all  --  lo     any     anywhere             anywhere
+   680 LOG        tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh LOG level warn prefix "FIREWALL_LOG: PERMIT "
-1135K ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED
+  110K ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh
-29996 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh
+   804 ACCEPT     all  --  lo     any     anywhere             anywhere
-     0 ACCEPT     all  --  lo     any     anywhere             anywhere
+11960 LOG        all  --  any    any     anywhere             anywhere             LOG level warn prefix "FIREWALL_LOG: DROP "
 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination
-Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
+Chain OUTPUT (policy ACCEPT 21944 packets, 5353K bytes)
  pkts bytes target     prot opt in     out     source               destination
-     0 ACCEPT     all  --  any    lo      anywhere             anywhere
+  1944 ACCEPT     all  --  any    lo      anywhere             anywhere
-     0 ACCEPT     all  --  any    lo      anywhere             anywhere
 root@logger02:~#
 </cli>
-Alright I have the basics running now but need to address a couple of things.  1) We are logging a ton of traffic like all of my SSH connectivity.  I need to tune this down and get into some rate limiting or things will get messy really quick.  2) Even though I have loaded RSYSLOG all logs are still going to the systemd logs.  I want to keep the system logs pointed to systemd (let it do its thing) and have all of the firewalls logs sent to /var/log for text based processing.
+===== The drama of logging in Linux =====
+I didn't know in this journey to consolidate logs to a centralized server I was going to end up blogging so much.   Turns out I needed to start another post to talk about the usage of systemd's journald vs rsyslog.  I went with rsyslog but you can [[blog:2025:0605_the_push_to_systemd_logs_and_how_its_breaking_my_heart|read the post]] if you want to know why.
+==== Going against the Linux Gods ====
+Now we need to shut off kernel logging in the journald. I go into details about this in the blog posting mentioned above, but I don't like the idea of cutting off kernel logs going to journald.  You're hands are kind of tied in this scenario and loading rsyslog takes you back to the legacy method of logging which isn't that bad.  Everything is in /var/log again.
+=== Install rsyslog and disable kernal logging in journald ===
+Basic rsyslog install is fine
+<cli>
+apt install rsyslog
+</cli>
+Here is a my /etc/rsyslog.conf file from the log servers.  This prevents FW logs from showing up in the kernel log and sends them to their own log file (easier to work with)
+<codeprism title=rsyslog.conf el=true hl=52-62>
+# /etc/rsyslog.conf configuration file for rsyslog
+#
+# For more information install rsyslog-doc and see
+# /usr/share/doc/rsyslog-doc/html/configuration/index.html
+#################
+#### MODULES ####
+#################
+module(load="imuxsock") # provides support for local system logging
+module(load="imklog")   # provides kernel logging support
+#module(load="immark")  # provides --MARK-- message capability
+# provides UDP syslog reception
+#module(load="imudp")
+#input(type="imudp" port="514")
+# provides TCP syslog reception
+#module(load="imtcp")
+#input(type="imtcp" port="514")
+###########################
+#### GLOBAL DIRECTIVES ####
+###########################
+#
+# Set the default permissions for all log files.
+#
+$FileOwner root
+$FileGroup adm
+$FileCreateMode 0640
+$DirCreateMode 0755
+$Umask 0022
+#
+# Where to place spool and state files
+#
+$WorkDirectory /var/spool/rsyslog
+#
+# Include all config files in /etc/rsyslog.d/
+#
+$IncludeConfig /etc/rsyslog.d/*.conf
+###############
+#### RULES ####
+###############
+# Forward messages with "FIREWALL_LOG:" prefix to a separate file
+:msg, contains, "FIREWALL_LOG:" /var/log/fw.log
+# This will forward all of the Firewall Logs to the syslog01 server from /etc/hosts over UDP 514
+:msg, contains, "FIREWALL_LOG:" @syslog01:514
+# Discard messages with the "FIREWALL_LOG:" prefix
+:msg, contains, "FIREWALL_LOG:" stop
+# Send all other messages to the journald socket - not sure this is needed I am doing it all in syslog now might remove later.
+*.* action(type="omuxsock" Socket="/run/systemd/journal/syslog")
+#
+# Log anything besides private authentication messages to a single log file
+#
+*.*;auth,authpriv.none		-/var/log/syslog
+#
+# Log commonly used facilities to their own log file
+#
+auth,authpriv.*			/var/log/auth.log
+cron.*				-/var/log/cron.log
+kern.*				-/var/log/kern.log
+mail.*				-/var/log/mail.log
+user.*				-/var/log/user.log
+#
+# Emergencies are sent to everybody logged in.
+#
+*.emerg				:omusrmsg:*
+</codeprism>
+Then edit /etc/systemd/journald.conf and uncomment and change the **ReadKMsg=yes** flag to **ReadKMsg=no**
+<cli>
+root@logger02:~# vim /etc/systemd/journald.conf
+</cli>
+<code title=journald.conf el=true hl=34>
+# Use 'systemd-analyze cat-config systemd/journald.conf' to display the full config.
+#
+# See journald.conf(5) for details.
+[Journal]
+#Storage=auto
+#Compress=yes
+#Seal=yes
+#SplitMode=uid
+#SyncIntervalSec=5m
+#RateLimitIntervalSec=30s
+#RateLimitBurst=10000
+#SystemMaxUse=
+#SystemKeepFree=
+#SystemMaxFileSize=
+#SystemMaxFiles=100
+#RuntimeMaxUse=
+#RuntimeKeepFree=
+#RuntimeMaxFileSize=
+#RuntimeMaxFiles=100
+#MaxRetentionSec=
+#MaxFileSec=1month
+ForwardToSyslog=yes
+#ForwardToKMsg=no
+#ForwardToConsole=no
+#ForwardToWall=yes
+#TTYPath=/dev/console
+#MaxLevelStore=debug
+MaxLevelSyslog=warn
+#MaxLevelKMsg=notice
+#MaxLevelConsole=info
+#MaxLevelWall=emerg
+#LineMax=48K
+ReadKMsg=no
+#Audit=no
+</code>
+Once you have rsyslog configured and logging disabled going to journald you need to restart both logging services.  Then you should start seeing firewall logs (fw.log) in your /var/log/ directory.
+<cli>
+systemctl restart systemd-journald.service
+systemctl restart rsyslog.service
+</cli>
+Alright I have the basics running now but need to address a couple of things.
+=== The Syslog Server - syslog01 ===
+FIXME
+Now you should see both servers "logger02 and logger01" in the log file on your syslog server.
+<file>
+-06-11T04:32:24+00:00 logger02 kernel: [514514.807037] FIREWALL_LOG: DROP IN=eth0 OUT= MAC=62:4b:80:04:8f:5d:fe:00:00:00:01:01:08:00 SRC=5.188.206.54 DST=209.38.71.201 LEN=40 TOS=0x00 PREC=0x00 TTL=235 ID=22090 PROTO=TCP SPT=8080 DPT=3344 WINDOW=1024 RES=0x00 SYN URGP=0
+-06-11T04:32:24+00:00 logger01 kernel: [ 3309.873056] FIREWALL_LOG: DROP IN=eth0 OUT= MAC=f6:5a:cb:13:e6:33:fe:00:00:00:01:01:08:00 SRC=79.124.8.112 DST=137.184.122.230 LEN=40 TOS=0x00 PREC=0x20 TTL=242 ID=54321 PROTO=TCP SPT=41609 DPT=34567 WINDOW=65535 RES=0x00 SYN URGP=0
+</file>
+  - <del>We are logging a ton of traffic like all of my SSH connectivity.  I need to tune this down and get into some rate limiting or things will get messy really quick.</del> **DONE**
+  - Even though I have loaded RSYSLOG all logs are still going to the systemd logs.  I want to keep the system logs pointed to systemd (let it do its thing) and have all of the firewalls logs sent to /var/log for text based processing. :!: I thought this was going to be an option, turns out log flexibility has gone away now that the world of Linux has push to systemd
 FIXME
@@ Line 81: / Line 252: @@
 By implementing this iptables policy, a firewall that allows SSH on port 22, denies all other inbound traffic, and logs both accepted and denied packets with the desired prefix can be created.
+[[:about|CONTACT ME:]] if you think I missed something, need to update anything, or you just have questions. Follow the contact rules or spam filters will get you.