The push to systemd logs and how its breaking my heart

I finally got myself started and focused (focus can be tough some days) on my side project TopTalkers I want to start keeping stats on devices that are scanning my networks, devices, honeypots, across the internet. As you can tell from my other blog post I figured out that the usage of rsyslog in Debian 12 has been depreciated in favor of sending all traffic to journald part of the systemd subsystem for logging.

Initially I was thinking this was better for Linux overall and maybe it will be eventually, but at this point the system doesn't make much sense to me. For years now I have learned to use journalctl -k to look into problems mostly at a system level or with some automated services that had issues. I have gotten used to it and journalctl was a nice way to scroll through the information when troubleshooting. Until I started this project, then things in journald got messy really quick.

From what I can tell journald is a big ass trash bucket of logs, just throw anything there and it holds on to it. After years and years of using rsyslog and knowing where to look for authentication issues, kernel issues, apache webserver issues, firewall/network issues, or just any service that wanted to log things it was broken out nicely in /var/log. You find the file for the service you want information on and open, read, grep, or tail the file for details on anything.

The problem here comes with iptables firewall logs. I imagine it would be the same thing for web server or load balancer logs as well. Admittedly most people probably aren't logging all of their connections/denies in iptables. I am probably a very specific edge case (wow it sucks when you realize you're the edge case), but any amount of iptables logs are now sent to the kernel log which is defaulted to journald. But when you send network logs to the journald process it gets really loaded really quick. I went through the steps of sending these logs from journald to rsyslog but there is no way to easily tell journald to drop them or ignore them from logging in the journald system. This means you will have to double your storage for logging network events.

For the time being the method I have found around this is to just disable kernel messaging in the /etc/systemd/journald.config file. You will need to remove the comment block on ReadKMsg=yes and change the yes to a no. You will then lose all kernel messaging in journald, I am not too worried about it they are now in /var/log/kernel.log.

ReadKMsg=no

Be sure to restart the journald service to apply the changes.

I think I going to go back to this method until the systemd/iptables folks figure out what they are doing. I have never been a big fan of Microsoft Windows Event viewer but even it seems to be better than the current state of logging in journald.