about | RSS Playground | Technical Projects and How Tos
Welcome to IGAZINE
Of all the websites in all the world you stumbled upon mine.
— Eric McWilliams 2025/05/08 18:48
I'm not sure what brought you here, be it random or some search engine, but welcome. I just wanted a place to put down random things in my head, day-to-day cool things I find, and anything I'd like to share with others.
The Blogs Starts here
This is just me thinking out loud.
The game of Monopoly vs The game of Life
If you think about it at the world scale do you think its possible we are playing things out like a game of monopoly? you keep playing this game going around the board over and over again and eventually someone has so much they win it all…
Another night in Franklin
I am in Franklin, TN this week and its nice, some times too nice, finding a good Mexican food place in Tennessee is one of the hardest quest you can put in front of a traveler. Besides jumping up in the middle of the night due to alerts from my EDR product alerting on a powershell false positive I feel like this trip has some potential to set some new direction for things at work. We have decided to move off of VMWare to Proxmox and improve our storage engineering by using ZFS. If I can get things started this will be a pretty interesting year or two ahead of me.
I may be putting my developer hat back on and hopefully putting some of this cybersecurity stuff behind me. I would love to build my own ERP system for a company but that is going to take a ton of debate and the breaking down of silos inside our company. If it doesn't happen I will just stick my head back in the cybersecurity space, enforce policy, and slap peoples hands while working my way to Bali.
Anonymous on American
I tired watching a documentary about Anonymous on my flight. I knew anything that was touching on Anonymous was going to touch on 4chan. I wasn't aware how much 4chan but even a little bit of /b content is way too much to have on a tablet screen in the middle of a commercial flight. I ended up pausing it and will pick back up when I get to the hotel tonight. I am not really sure if there is much they can put in an Anonymous documentary I don't already know but it was a pretty amazing time in the internet and I was glad to be whiteness to it all.
The push to systemd logs and how its breaking my heart
I finally got myself started and focused (focus can be tough some days) on my side project TopTalkers I want to start keeping stats on devices that are scanning my networks, devices, honeypots, across the internet. As you can tell from my other blog post I figured out that the usage of rsyslog in Debian 12 has been depreciated in favor of sending all traffic to journald part of the systemd subsystem for logging.
Initially I was thinking this was better for Linux overall and maybe it will be eventually, but at this point the system doesn't make much sense to me. For years now I have learned to use journalctl -k to look into problems mostly at a system level or with some automated services that had issues. I have gotten used to it and journalctl was a nice way to scroll through the information when troubleshooting. Until I started this project, then things in journald got messy really quick.
From what I can tell journald is a big ass trash bucket of logs, just throw anything there and it holds on to it. After years and years of using rsyslog and knowing where to look for authentication issues, kernel issues, apache webserver issues, firewall/network issues, or just any service that wanted to log things it was broken out nicely in /var/log. You find the file for the service you want information on and open, read, grep, or tail the file for details on anything.
The problem here comes with iptables firewall logs. I imagine it would be the same thing for web server or load balancer logs as well. Admittedly most people probably aren't logging all of their connections/denies in iptables. I am probably a very specific edge case (wow it sucks when you realize you're the edge case), but any amount of iptables logs are now sent to the kernel log which is defaulted to journald. But when you send network logs to the journald process it gets really loaded really quick. I went through the steps of sending these logs from journald to rsyslog but there is no way to easily tell journald to drop them or ignore them from logging in the journald system. This means you will have to double your storage for logging network events.
For the time being the method I have found around this is to just disable kernel messaging in the /etc/systemd/journald.config file. You will need to remove the comment block on ReadKMsg=yes and change the yes to a no. You will then lose all kernel messaging in journald, I am not too worried about it they are now in /var/log/kernel.log.
ReadKMsg=no
Be sure to restart the journald service to apply the changes.
I think I going to go back to this method until the systemd/iptables folks figure out what they are doing. I have never been a big fan of Microsoft Windows Event viewer but even it seems to be better than the current state of logging in journald.
Debian 12 logging change and my syslog project
Picking back up from my technical posting on configuring centralized syslog this is what my logging directory looks like in Debian 12 (Bookworm). Things have gotten interesting since the last time I started a syslog project.
root@syslog01:/var/log# ls README cloud-init-output.log exim4 runit alternatives.log cloud-init.log journal unattended-upgrades apt dpkg.log lastlog wtmp btmp droplet-agent.update.log private
Well it looks like a lot of things have changed in the logging world since Debian 9. I know people have had their political arguments over time but looks like systemd has finally taken over logging as well. I have never really had a dog in the fight and just kind of took to systemd as it took over the Linux world. Here is the README content.
root@syslog01:/var/log# less README You are looking for the traditional text log files in /var/log, and they are gone? Here's an explanation on what's going on: You are running a systemd-based OS where traditional syslog has been replaced with the Journal. The journal stores the same (and more) information as classic syslog. To make use of the journal and access the collected log data simply invoke "journalctl", which will output the logs in the identical text-based format the syslog files in /var/log used to be. For further details, please refer to journalctl(1). Alternatively, consider installing one of the traditional syslog implementations available for your distribution, which will generate the classic log files for you. Syslog implementations such as syslog-ng or rsyslog may be installed side-by-side with the journal and will continue to function the way they always did. Thank you! Further reading: man:journalctl(1) man:systemd-journald.service(8) man:journald.conf(5) https://0pointer.de/blog/projects/the-journal.html README (END)
Now after some reading I believe I will keep using rsyslog or syslog-ng (the debate will rage on during this project). I think some users have legit concerns with corruption and journald.