technical:remote_syslog_setup

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
technical:remote_syslog_setup [2025/06/04 00:17] – super_stundertechnical:remote_syslog_setup [2025/06/11 04:45] (current) – [Going against the Linux Gods] super_stunder
Line 1: Line 1:
 +====== Remote Syslog - IPTables - Tigers and Bears OH MY ======
 +
 +
 I have been working on a simple SYSLOG project to gather scanning data off of host around the internet and centralize it.  The no telling what I will do with the data at that point.  Its a project I thought of a while back and I nicknamed it TopTalkers.  Basically this is the setup I am going for, very simple a remote syslog server with two host sending it traffic.   I have been working on a simple SYSLOG project to gather scanning data off of host around the internet and centralize it.  The no telling what I will do with the data at that point.  Its a project I thought of a while back and I nicknamed it TopTalkers.  Basically this is the setup I am going for, very simple a remote syslog server with two host sending it traffic.  
  
 <graphviz> <graphviz>
 digraph MyGraph { digraph MyGraph {
-  logger01 -> syslog01 +  syslog01 -> logger01 [dir=back]; 
-  logger02 -> syslog01+  syslog01 -> logger02 [dir=back]; 
 +  label = "Basic Concept"
 } }
 </graphviz> </graphviz>
Line 13: Line 17:
 I had worked on a project like this a couple of years/versions of Debian ago and it was always a debate between rsyslog and syslog-ng.  After setting up the three servers I logged into the syslog01 server and went into /var/log/ and things looked different.  First off there was a README file in the directory! I had worked on a project like this a couple of years/versions of Debian ago and it was always a debate between rsyslog and syslog-ng.  After setting up the three servers I logged into the syslog01 server and went into /var/log/ and things looked different.  First off there was a README file in the directory!
  
-I am going to move the sidebar conversation to the blog instead of taking up this whole technical discussion with journald vs syslog debate.+I am going to move the sidebar conversation to the [[blog:2025:0604_debian_12_logging_change_and_my_syslog_project|blog]] instead of taking up this whole technical discussion with journald vs syslog debate. Alright lets get back to some remote logging shall we???
  
-<code> +---- 
-root@syslog01:/var/log# ls  +===== Configuring iptables (the hell of logging denies) =====
-README   cloud-init-output.log     exim4    runit +
-alternatives.log  cloud-init.log     journal  unattended-upgrades +
-apt   dpkg.log     lastlog  wtmp +
-btmp   droplet-agent.update.log  private +
-</code>+
  
-Well it looks like a lot of things have changed in the logging world since Debian 9.  I know people have had their [[https://en.wikipedia.org/wiki/Systemd#History|political arguments]] over time but looks like systemd has finally taken over logging as well.  I have never really had a dog in the fight and just kind of took to systemd as it took over the Linux world. Here is the README content. 
  
-<code> +<del>So I am going to start this off and load rsyslog and UFW on the logger servers then I will go from there. I am using UFW for basic firewall configs it makes things way easier when building basic rules.
-root@syslog01:/var/log# less README +
-You are looking for the traditional text log files in /var/log, and they are +
-gone?+
  
-Here's an explanation on what's going on:+<cli>root@logger01:~# apt install rsyslog ufw</cli></del>
  
-You are running a systemd-based OS where traditional syslog has been replaced +I started this with the plans of using UFW to keep it simple things got pretty detailed pretty quick so I ended up switching directly to iptables.  I may end up doing another install with netfilter to keep up with the times.
-with the Journal. The journal stores the same (and more) information as classic +
-syslog. To make use of the journal and access the collected log data simply +
-invoke "journalctl", which will output the logs in the identical text-based +
-format the syslog files in /var/log used to be. For further details, please +
-refer to journalctl(1).+
  
-Alternatively, consider installing one of the traditional syslog +Alright so now I am actually going to use a little bit of help from Google Gemini.  At one time in life I loved spending the whole afternoon messing with [[https://www.netfilter.org/|iptables]] rules these days I am just going to ask Google and let Gemini kick it out.  A bit of warning here the first method/policy Gemini gave me locked me out of the system on the first line. The firewall rules are only temporary and are lost on reboot unless you load the "iptables-persistent" package after you have things applied and tested.  
-implementations available for your distribution, which will generate the +
-classic log files for you. Syslog implementations such as syslog-ng or rsyslog +
-may be installed side-by-side with the journal and will continue to function +
-the way they always did.+
  
-Thank you!+<cli> 
 +-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
 +-A INPUT -p tcp -m tcp --dport 22 -j LOG --log-prefix "FIREWALL_LOG: PERMIT " 
 +-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT 
 +-A INPUT -i lo -j ACCEPT 
 +-A INPUT -j LOG --log-prefix "FIREWALL_LOG: DROP " 
 +-A OUTPUT -o lo -j ACCEPT 
 +</cli>
  
-Further reading: +==== Saving iptables config and setting persistence after reboots ==== 
-        man:journalctl(1) + 
-        man:systemd-journald.service(8) +After you have tested access back into your device over SSH you can now issue the persistent command to maintain the rules at startup. Turns on iptables-persistent isn't installed by default.  So install it then run it and accept the warning to save the current policy.  
-        man:journald.conf(5) + 
-        https://0pointer.de/blog/projects/the-journal.html +<cli> 
-README (END)+apt install iptables-persistent 
 +</cli> 
 + 
 +After iptables-persistent is install you can now run iptables-save to your policy file and it will load again after reboots. 
 + 
 +<cli> 
 +root@logger02:~# iptables-save > /etc/iptables/rules.v4 
 +</cli> 
 + 
 +==== Displaying iptables rules ==== 
 +While digging into the rules I was just using the iptables -L command to list the policy.  This command gives some basic down and dirty output and I have found using iptabels -L -v with the verbose option gives much better results with a packet counter and the interface information that is left out of the basic -L command. 
 + 
 + 
 +<cli>root@logger02:~# iptables -L -v 
 +Chain INPUT (policy DROP 14136 packets, 706K bytes) 
 + pkts bytes target     prot opt in     out     source               destination          
 +19294 4901K ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED 
 +   12   680 LOG        tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh LOG level warn prefix "FIREWALL_LOG: PERMIT " 
 + 1250  110K ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh 
 +   12   804 ACCEPT     all  --  lo     any     anywhere             anywhere             
 +  247 11960 LOG        all  --  any    any     anywhere             anywhere             LOG level warn prefix "FIREWALL_LOG: DROP " 
 + 
 +Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) 
 + pkts bytes target     prot opt in     out     source               destination          
 + 
 +Chain OUTPUT (policy ACCEPT 21944 packets, 5353K bytes) 
 + pkts bytes target     prot opt in     out     source               destination          
 +   24  1944 ACCEPT     all  --  any    lo      anywhere             anywhere             
 +root@logger02:~#  
 +  
 +</cli> 
 + 
 +===== The drama of logging in Linux ===== 
 +I didn't know in this journey to consolidate logs to a centralized server I was going to end up blogging so much.   Turns out I needed to start another post to talk about the usage of systemd's journald vs rsyslog.  I went with rsyslog but you can [[blog:2025:0605_the_push_to_systemd_logs_and_how_its_breaking_my_heart|read the post]] if you want to know why.  
 + 
 +==== Going against the Linux Gods ==== 
 +Now we need to shut off kernel logging in the journald. I go into details about this in the blog posting mentioned above, but I don't like the idea of cutting off kernel logs going to journald.  You're hands are kind of tied in this scenario and loading rsyslog takes you back to the legacy method of logging which isn't that bad.  Everything is in /var/log again. 
 + 
 +=== Install rsyslog and disable kernal logging in journald === 
 +Basic rsyslog install is fine 
 +<cli> 
 +apt install rsyslog 
 +</cli> 
 + 
 +Here is a my /etc/rsyslog.conf file from the log servers.  This prevents FW logs from showing up in the kernel log and sends them to their own log file (easier to work with) 
 +<codeprism title=rsyslog.conf el=true hl=52-62> 
 +# /etc/rsyslog.conf configuration file for rsyslog 
 +# 
 +# For more information install rsyslog-doc and see 
 +# /usr/share/doc/rsyslog-doc/html/configuration/index.html 
 + 
 + 
 +################# 
 +#### MODULES #### 
 +################# 
 + 
 +module(load="imuxsock") # provides support for local system logging 
 +module(load="imklog")   # provides kernel logging support 
 +#module(load="immark")  # provides --MARK-- message capability 
 + 
 +# provides UDP syslog reception 
 +#module(load="imudp") 
 +#input(type="imudp" port="514") 
 + 
 +# provides TCP syslog reception 
 +#module(load="imtcp") 
 +#input(type="imtcp" port="514") 
 + 
 + 
 +########################### 
 +#### GLOBAL DIRECTIVES #### 
 +########################### 
 + 
 +# 
 +# Set the default permissions for all log files. 
 +# 
 +$FileOwner root 
 +$FileGroup adm 
 +$FileCreateMode 0640 
 +$DirCreateMode 0755 
 +$Umask 0022 
 + 
 +# 
 +# Where to place spool and state files 
 +# 
 +$WorkDirectory /var/spool/rsyslog 
 + 
 +# 
 +# Include all config files in /etc/rsyslog.d/ 
 +# 
 +$IncludeConfig /etc/rsyslog.d/*.conf 
 + 
 + 
 +############### 
 +#### RULES #### 
 +############### 
 + 
 +# Forward messages with "FIREWALL_LOG:" prefix to a separate file 
 +:msg, contains, "FIREWALL_LOG:" /var/log/fw.log 
 + 
 +# This will forward all of the Firewall Logs to the syslog01 server from /etc/hosts over UDP 514 
 +:msg, contains, "FIREWALL_LOG:" @syslog01:514 
 + 
 +# Discard messages with the "FIREWALL_LOG:" prefix 
 +:msg, contains, "FIREWALL_LOG:" stop 
 + 
 +# Send all other messages to the journald socket - not sure this is needed I am doing it all in syslog now might remove later. 
 +*.* action(type="omuxsock" Socket="/run/systemd/journal/syslog") 
 + 
 +# 
 +# Log anything besides private authentication messages to a single log file 
 +# 
 +*.*;auth,authpriv.none -/var/log/syslog 
 + 
 +# 
 +# Log commonly used facilities to their own log file 
 +# 
 +auth,authpriv.* /var/log/auth.log 
 +cron.* -/var/log/cron.log 
 +kern.* -/var/log/kern.log 
 +mail.* -/var/log/mail.log 
 +user.* -/var/log/user.log 
 + 
 +# 
 +# Emergencies are sent to everybody logged in. 
 +# 
 +*.emerg :omusrmsg:* 
 +</codeprism> 
 + 
 + 
 +Then edit /etc/systemd/journald.conf and uncomment and change the **ReadKMsg=yes** flag to **ReadKMsg=no** 
 + 
 +<cli> 
 +root@logger02:~# vim /etc/systemd/journald.conf  
 +</cli> 
 + 
 +<code title=journald.conf el=true hl=34> 
 +# Use 'systemd-analyze cat-config systemd/journald.conf' to display the full config. 
 +# 
 +# See journald.conf(5) for details. 
 + 
 +[Journal] 
 +#Storage=auto 
 +#Compress=yes 
 +#Seal=yes 
 +#SplitMode=uid 
 +#SyncIntervalSec=5m 
 +#RateLimitIntervalSec=30s 
 +#RateLimitBurst=10000 
 +#SystemMaxUse= 
 +#SystemKeepFree= 
 +#SystemMaxFileSize= 
 +#SystemMaxFiles=100 
 +#RuntimeMaxUse= 
 +#RuntimeKeepFree= 
 +#RuntimeMaxFileSize= 
 +#RuntimeMaxFiles=100 
 +#MaxRetentionSec= 
 +#MaxFileSec=1month 
 +ForwardToSyslog=yes 
 +#ForwardToKMsg=no 
 +#ForwardToConsole=no 
 +#ForwardToWall=yes 
 +#TTYPath=/dev/console 
 +#MaxLevelStore=debug 
 +MaxLevelSyslog=warn 
 +#MaxLevelKMsg=notice 
 +#MaxLevelConsole=info 
 +#MaxLevelWall=emerg 
 +#LineMax=48K 
 +ReadKMsg=no 
 +#Audit=no
 </code> </code>
  
-Now after some reading I believe I will keep using rsyslog or syslog-ng (the debate will rage on during this project).  I think some users have legit [[https://www.reddit.com/r/linux/comments/1y6q0l/systemds_binary_logs_and_corruption/|concerns]] with corruption and journald. +Once you have rsyslog configured and logging disabled going to journald you need to restart both logging services.  Then you should start seeing firewall logs (fw.log) in your /var/log/ directory. 
 + 
 +<cli> 
 +systemctl restart systemd-journald.service 
 +systemctl restart rsyslog.service 
 +</cli> 
 + 
 + 
 + 
 + 
 +Alright I have the basics running now but need to address a couple of things.   
 + 
 +=== The Syslog Server - syslog01 === 
 +FIXME 
 + 
 +Now you should see both servers "logger02 and logger01" in the log file on your syslog server.  
 +<file> 
 +2025-06-11T04:32:24+00:00 logger02 kernel: [514514.807037] FIREWALL_LOG: DROP IN=eth0 OUT= MAC=62:4b:80:04:8f:5d:fe:00:00:00:01:01:08:00 SRC=5.188.206.54 DST=209.38.71.201 LEN=40 TOS=0x00 PREC=0x00 TTL=235 ID=22090 PROTO=TCP SPT=8080 DPT=3344 WINDOW=1024 RES=0x00 SYN URGP=0  
 +2025-06-11T04:32:24+00:00 logger01 kernel: [ 3309.873056] FIREWALL_LOG: DROP IN=eth0 OUT= MAC=f6:5a:cb:13:e6:33:fe:00:00:00:01:01:08:00 SRC=79.124.8.112 DST=137.184.122.230 LEN=40 TOS=0x00 PREC=0x20 TTL=242 ID=54321 PROTO=TCP SPT=41609 DPT=34567 WINDOW=65535 RES=0x00 SYN URGP=0 
 +</file> 
 + 
 +  - <del>We are logging a ton of traffic like all of my SSH connectivity.  I need to tune this down and get into some rate limiting or things will get messy really quick.</del> **DONE** 
 +  - Even though I have loaded RSYSLOG all logs are still going to the systemd logs.  I want to keep the system logs pointed to systemd (let it do its thing) and have all of the firewalls logs sent to /var/log for text based processing. :!: I thought this was going to be an option, turns out log flexibility has gone away now that the world of Linux has push to systemd 
 + 
 +FIXME 
 +Important considerations: 
 + 
 +Order of rules: The LOG rule must be at the very beginning of the chain to log all packets. 
 +Log level: Adjust the --log-level to suit your needs and logging system. Levels range from 0 (highest severity) to 7 (debug). 
 +Rate limiting: Logging all packets can generate a lot of log data. Use the --limit and --limit-burst options to rate-limit logging if needed. 
 + 
 +By implementing this iptables policy, a firewall that allows SSH on port 22, denies all other inbound traffic, and logs both accepted and denied packets with the desired prefix can be created. 
 + 
 +[[:about|CONTACT ME:]] if you think I missed something, need to update anything, or you just have questions. Follow the contact rules or spam filters will get you.  
 +  
 + 
 + 
  • technical/remote_syslog_setup.1748996223.txt.gz
  • Last modified: 2025/06/04 00:17
  • by super_stunder