I had setup rsyslog and alloy + loki to consolidate logs from all my servers in my home network for analysis, graphing and alerting. My goal was to:<\/p>\n\n\n\n
After some struggle, I was able to set this up with a combination of Dynafile and RSYSLOG_SyslogProtocol23Format in the loki forward format. It mostly worked – the keyword is mostly. <\/p>\n\n\n\n
Some of the issues I wanted to fix:<\/p>\n\n\n\n
Let’s begin the madness. We start first with a highly embarrassing mistake from me – I simply added to \/etc\/rsyslog.conf:<\/p>\n\n\n\n
*.* @@logserver:514<\/code><\/pre>\n\n\n\nAnd all hell broke loose. Guess what I just did – I setup an infinite loop of log lines. Since remote files send logs through the same interface and I just setup a forward rule for all log lines, we started an insane amount of writes by sending the same logs over and over again in a never-ending, ever-growing cascade of log lines.<\/p>\n\n\n\n
Ok – obviously that was not very smart. Let’s make sure we create a ruleset that processes remote log lines and then add the forwarding rule which will then only forward local log lines.<\/p>\n\n\n\n
ruleset(\n name=\"RS_Remote\"\n queue.type=\"LinkedList\" # In-memory queue\n queue.size=\"100000\" # Size appropriate for this action\n queue.dequeueBatchSize=\"1024\" # Fine-tune batching for IOPS\n queue.minDequeueBatchSize=\"500\" # minimum msgs before dequeue\n) {\n if $programname == \"\" then {\n action(\n type=\"omfile\"\n dynaFile=\"RemoteLogsWithoutProgram\"\n ) \n } else {\n action(\n type=\"omfile\" \n dynaFile=\"RemoteLogs\"\n ) \n } \n}<\/code><\/pre>\n\n\n\nOk – that stopped the endless log spew, but it completely broke the working dynaFile setup (and loki appname) where it was working most of the time. Suddenly, $programname<\/code> is always empty once I made this change. For reference, here’s the dynaFile setup that worked prior to making this a ruleset based processing:<\/p>\n\n\n\ntemplate(name=\"RemoteLogs\" type=\"string\" string=\"\/storage\/logs\/remote\/%HOSTNAME%\/%PROGRAMNAME%.log\")\ntemplate(name=\"RemoteLogsWithoutProgram\" type=\"string\" string=\"\/storage\/logs\/remote\/%HOSTNAME%\/default.log\")\n\nif $programname == \"\" then {\n action(type=\"omfile\" dynaFile=\"RemoteLogsWithoutProgram\")\n} else {\n action(type=\"omfile\" dynaFile=\"RemoteLogs\")\n}<\/code><\/pre>\n\n\n\nNote that nothing really changed other than moving the if\/else into a ruleset and attaching that ruleset to my imudp and imtcp inputs. And now, our steep descent into the madness that is rsyslog begins for real.<\/p>\n\n\n\n
After several abortive hours of search + AI fu (AI assistants are absolutely horrible with anything complex related to rsyslog) and just blindly trying to get things working without putting in any effort to actually understand rsyslog, I gave up that approach. Clearly, rsyslog, like ed (the king of editors), was meant for people who knew what they were doing and had the attention span longer than a 6 month old kitten.<\/p>\n\n\n\n
After spending more time than I care to admit reading the simultaneously sufficient yet difficult to navigate + read + understand documentation of rsyslog, here’s my glorious solution. This is probably the most complex example of a rsyslog configuration on the web that you can find – I am pretty sure there are more complex setups being run, but no one probably wants to admit to it in public \ud83d\ude42<\/p>\n\n\n\n
# \/etc\/rsyslog.conf configuration file for rsyslog\n#\n# For more information install rsyslog-doc and see\n# \/usr\/share\/doc\/rsyslog-doc\/html\/configuration\/index.html\n\n\n#################\n#### MODULES ####\n#################\n\n#module(load=\"imuxsock\") # provides support for local system logging\n#module(load=\"imklog\") # provides kernel logging support\n#module(load=\"immark\") # provides --MARK-- message capability\n\n$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat\n\n\nruleset(name=\"PROGRAMNAME_FIXER\") {\n\n set $.programname = \"\";\n set $.procid = \"\";\n\n if $programname == \"\" then {\n if re_match($syslogtag, '^[^[]+\\\\[[0-9]+\\\\]') then { \n set $.programname = re_extract($syslogtag, '^([^[]+)', 0, 1, \"\");\n set $.procid = re_extract($syslogtag, '\\\\[([0-9]+)\\\\]', 0, 1, \"\");\n }\n else if re_match($syslogtag, '^\\\\[.*\\\\]$' ) then {\n set $.programname = re_extract($syslogtag, '^\\\\[(.*)\\\\]$', 0, 1, \"\");\n } else {\n set $.syslogtag = field($msg, 32, 1);\n if $.syslogtag == \"\" then {\n set $.syslogtag = field($msg, 32, 2);\n }\n if re_match($.syslogtag, '^[^[]+\\\\[[0-9]+\\\\]') then {\n set $.programname = re_extract($.syslogtag, '^([^[]+)', 0, 1, \"\");\n set $.procid = re_extract($.syslogtag, '\\\\[([0-9]+)\\\\]', 0, 1, \"\");\n }\n else if re_match($syslogtag, ':') and re_match($syslogtag, '^[^,+]+$') then {\n set $.programname = re_extract($syslogtag, '([^:]+)', 0, 1, \"\");\n }\n else if re_match($.syslogtag, '[a-zA-Z][a-zA-Z_0-9-]+:') then {\n set $.programname = re_extract($.syslogtag, '([a-zA-Z_0-9-]+)', 0, 1, \"\");\n }\n }\n }\n\n if $.programname == \"\" then {\n set $.programname = \"-\";\n }\n if $.procid == \"\" then {\n set $.procid = \"-\";\n }\n\n set $!custom!programname = $.programname;\n set $!custom!procid = $.procid;\n}\n\ntemplate(name=\"DBG\" type=\"list\") {\n constant(value=\"' syslogtag: '\") property(name=\"syslogtag\")\n constant(value=\"' custom-program-name: '\") property(name=\"$!custom!programname\")\n constant(value=\"' custom-proc-id: '\") property(name=\"$!custom!procid\")\n constant(value=\"' msg: '\") property(name=\"msg\" controlCharacters=\"escape\")\n constant(value=\"' rawmsg: '\") property(name=\"rawmsg\" controlCharacters=\"escape\")\n constant(value=\"DBG: FROMHOST: '\") property(name=\"fromhost\")\n constant(value=\"' inputname: '\") property(name=\"inputname\")\n constant(value=\"' time reported: '\") property(name=\"timereported\" dateFormat=\"rfc3339\")\n constant(value=\"' hostname: '\") property(name=\"hostname\")\n constant(value=\"'\\n\")\n}\n\ntemplate(name=\"RemoteLogs\" type=\"string\" string=\"\/storage\/logs\/remote\/%HOSTNAME%\/%PROGRAMNAME%.log\")\ntemplate(name=\"RemoteLogsWithoutProgram\" type=\"string\" string=\"\/storage\/logs\/remote\/%HOSTNAME%\/default.log\")\n\ntemplate(name=\"RFC5424_CustomRobustList\" type=\"list\") {\n # 1. PRI and VERSION\n constant(value=\"<\")\n property(name=\"pri\")\n constant(value=\">1 \")\n\n # 2. TIMESTAMP\n property(name=\"timereported\" dateFormat=\"rfc3339\")\n constant(value=\" \")\n\n # 3. HOSTNAME\n property(name=\"hostname\")\n constant(value=\" \")\n\n # 4. APP-NAME (Using custom property, outputs '-' if empty)\n property(name=\"$!custom!programname\" controlCharacters=\"space\")\n constant(value=\" \")\n\n # 5. PROCID (Using custom property, outputs '-' if empty)\n property(name=\"$!custom!procid\" controlCharacters=\"space\")\n constant(value=\" \")\n\n # 6. MSGID (Outputs '-' if empty)\n property(name=\"msgid\" controlCharacters=\"space\")\n constant(value=\" \")\n\n # 7. STRUCTURED-DATA (Outputs '-' if empty)\n constant(value=\"-\")\n #property(name=\"structureddata\")\n\n # 8. MESSAGE\n constant(value=\" \")\n property(name=\"msg\")\n constant(value=\"\\n\")\n}\n\nruleset(\n name=\"RS_Remote\"\n queue.type=\"LinkedList\" # In-memory queue\n queue.size=\"100000\" # Size appropriate for this action\n queue.dequeueBatchSize=\"1024\" # Fine-tune batching for IOPS\n queue.minDequeueBatchSize=\"500\" # minimum msgs before dequeue\n) {\n call PROGRAMNAME_FIXER\n\n if $programname == \"\" then {\n action(\n type=\"omfile\"\n dynaFile=\"RemoteLogsWithoutProgram\"\n )\n } else {\n action(\n type=\"omfile\" \n dynaFile=\"RemoteLogs\"\n )\n }\n\n if $!custom!programname == \"-\" then {\n action(type=\"omfile\" file=\"\/storage\/logs\/remote\/log\/rsyslog-debug.log\" template=\"DBG\")\n }\n\n action(\n type=\"omfwd\"\n target=\"loki.lan\"\n port=\"1514\"\n protocol=\"tcp\"\n template=\"RFC5424_CustomRobustList\"\n )\n}\n\n## provides UDP syslog reception\nmodule(load=\"imudp\")\ninput(\n type=\"imudp\" \n port=\"514\" \n address=\"<ip>\" \n rateLimit.Interval=\"600\" \n rateLimit.Burst=\"1000\" \n ruleSet=\"RS_Remote\"\n)\n\n# provides TCP syslog reception\nmodule(load=\"imtcp\")\ninput(\n type=\"imtcp\" \n port=\"514\" \n address=\"<ip>\" \n rateLimit.Interval=\"600\" \n rateLimit.Burst=\"10000\" \n ruleSet=\"RS_Remote\"\n)\n\n\n###########################\n#### GLOBAL DIRECTIVES ####\n###########################\n\n#\n# Set the default permissions for all log files.\n#\n$FileOwner syslog\n$FileGroup adm\n$FileCreateMode 0640\n$DirCreateMode 0755\n$Umask 0022\n\n#\n# Where to place spool and state files\n#\n$WorkDirectory \/storage\/logs\/spool\/rsyslog\n\n# Main queue: in-RAM linked list with large batched dequeues\nmain_queue(\n queue.type=\"LinkedList\"\n queue.size=\"200000\" # total msgs in RAM (tune to RAM)\n queue.highWatermark=\"150000\"\n queue.lowWatermark=\"50000\"\n\n # Batch processing to reduce small writes; trades a bit of latency for IOPS\n queue.dequeueBatchSize=\"1000\" # max msgs per dequeue\n queue.minDequeueBatchSize=\"500\" # minimum msgs before dequeue\n)\n\n# Use traditional timestamp format\n\n# Drop to lower privilege\n$PrivDropToUser syslog\n$PrivDropToGroup syslog\n\n###############\n#### RULES ####\n###############\n\n#\n# Log anything besides private authentication messages to a single log file\n#\n#*.*;auth,authpriv.none -\/storage\/logs\/syslog\n\n#\n# Log commonly used facilities to their own log file\n#\n#auth,authpriv.* \/storage\/logs\/auth.log\n#cron.* -\/storage\/logs\/cron.log\n#kern.* -\/storage\/logs\/kern.log\n#mail.* -\/storage\/logs\/mail.log\n#user.* -\/storage\/logs\/user.log\n\n#\n# Emergencies are sent to everybody logged in.\n#\n#*.emerg :omusrmsg:*\n\n#*.* -?RemoteLogs\n\n\n# BEGIN ANSIBLE MANAGED BLOCK: remote log forwarding\n*.* @@<ip>:514\n# END ANSIBLE MANAGED BLOCK: remote log forwarding<\/code><\/pre>\n\n\n\nNote that in this config, while Loki gets structured appname, I haven’t segmented the stored log files via omfwd yet. I am still debating whether I want to do this or not – it should be fairly straightforward since I do have the $!custom!programname<\/code> set up already.<\/p>\n\n\n\nA lot of the insanity is in the PROGRAMNAME_FIXER<\/code> ruleset – this ruleset uses a bunch of regex munging to handle all the different types of log spew (Unifi in particular spews so many interesting varieties of logs) so we can extract a meaningful appname<\/code> (and possibly procid<\/code>) for loki.<\/p>\n\n\n\nSome important and hopefully useful tips to work with RSyslog Configuration, Alloy and Loki:<\/p>\n\n\n\n
\n- Loki syslog receiver that you configure in Alloy by default accepts RFC5424 format input.<\/li>\n\n\n\n
- RFC5424 expects the following fields:\n
\nPRI - Priority<\/code><\/li>\n\n\n\nVER - Version<\/code><\/li>\n\n\n\nspace<\/code><\/li>\n\n\n\nTIMESTAMP - RFC3339 format<\/code><\/li>\n\n\n\nspace<\/code><\/li>\n\n\n\nhostname<\/code><\/li>\n\n\n\nspace<\/code><\/li>\n\n\n\nappname<\/code><\/li>\n\n\n\nspace<\/code><\/li>\n\n\n\nprocid<\/code><\/li>\n\n\n\nspace<\/code><\/li>\n\n\n\nmsgid<\/code><\/li>\n\n\n\nspace<\/code><\/li>\n\n\n\nstructured data<\/code><\/li>\n\n\n\nspace<\/code><\/li>\n\n\n\nmsg<\/code><\/li>\n<\/ul>\n<\/li>\n\n\n\n- If any field is empty, it should be represented by
'-'<\/code><\/li>\n\n\n\n- In RSyslog config file language (aka RainerScript)\n
\n$!<\/code> variables are “global” message level variables. They are available once set elsewhere later in the message processing pipeline.<\/li>\n\n\n\n$.<\/code> variables are “local” variables. They are only available in the context of that ruleset or processing entity in the message processing pipeline.<\/li>\n\n\n\n- Templates are how you control output. You can only use message level properties or constants in templates.<\/li>\n\n\n\n
- Rulesets are how you can attach a processing pipe to an input or process a message further. Rulesets can call each other. A ruleset can have multiple actions. Actions can be conditional.<\/li>\n\n\n\n
- You can do printf debugging by using a combination of a debug template, setting arbitrary message variables which are used by the debug template and then an output omfile action with the debug template. Now this file will spew out your glorious printf debugging for RainerScript.<\/li>\n\n\n\n
- The Regex dialect used in RainerScript is Posix Extended Regular Expressions<\/a>.<\/li>\n\n\n\n
set<\/code> statements need a ;<\/code> at the end. I can’t remember how many times I kept forgetting that!<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\nFinally, let me re-emphasize: Every single AI assistant was beyond useless in this exercise and made it a lot more frustrating that it needed to be. They did not understand any single important information listed in the bullets above about RainerScript and kept driving me insane with their plausible but completely incorrect guidance. Hopefully, now that this blog post is out, the agents will learn and become better at RainerScript?<\/p>\n","protected":false},"excerpt":{"rendered":"
I had setup rsyslog and alloy + loki to consolidate logs from all my servers in my home network for analysis, graphing and alerting. My goal was to: After some struggle, I was able to set this up with a combination of Dynafile and RSYSLOG_SyslogProtocol23Format in the loki forward format. It mostly worked – the…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-601","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts\/601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/comments?post=601"}],"version-history":[{"count":2,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts\/601\/revisions"}],"predecessor-version":[{"id":603,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts\/601\/revisions\/603"}],"wp:attachment":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/media?parent=601"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/categories?post=601"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/tags?post=601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}