In my case, I chose the following setup:<\/p>\n\n\n\n
- \n
- Every node runs rsyslog<\/a> and sends the logs to a central rsyslog server. You can alternatively use alloy to send the logs to a centralized location as well.<\/li>\n\n\n\n
- The central log server (also running rsyslog and receiving logs from all nodes), saves them to files and forwards them to an alloy<\/a> host. The rsyslog server processes the logs to add necessary fields that are helpful in queries – you can read more about how this was setup here.<\/a><\/li>\n\n\n\n
- The alloy<\/a> endpoint processes the logs to sanitize labels, drop high cardinality labels and forwards them to a loki<\/a> instance running on the same host.<\/li>\n\n\n\n
- The loki<\/a> instance is connected as a data source to a grafana endpoint. Grafana<\/a> is used for creating dashboards and authoring queries against the loki instance with all the centralized logs.<\/li>\n<\/ul>\n\n\n\n
Here’s a visual diagram of the system with various components and how they connect to each other.<\/p>\n\n\n\n
![\"\"](\"https:\/\/nramkumar.org\/tech\/wp-content\/uploads\/2025\/11\/Untitled-diagram-2025-11-27-044543-1024x693.png\")
<\/figure>\n\n\n\nSome key things to keep in mind when you set this up:<\/p>\n\n\n\n
- \n
- Loki works on the concept of labels – labels allow you to narrow down the search scope for a log query. Examples of labels can be hostname, appname etc. Labels are very useful in creating efficient queries through logs across all your hosts.<\/li>\n\n\n\n
- Loki handles indexed labels by creating individual files for each unique combination of label sets.<\/li>\n\n\n\n
- What that means is that if you have high cardinality labels (a label with a large number of unique values), you will quickly run out of inode space on your loki host due to the large number of small files that it will create.<\/li>\n\n\n\n
- You can use logcli to find all your loki labels and understand the cardinality of your existing labels with this command: \n
- \n
export LOKI_ADDR=\"http:\/\/127.0.0.1:3100\"; .\/logcli-linux-amd64 series '{}' --analyze-labels --since=1h<\/code><\/li>\n<\/ul>\n<\/li>\n\n\n\n- You can also check how much disk space you are using and how much inode space you are using (note that you may run out of inode space even though you have plenty of disk space) using the
df<\/code> or du commands.<\/li>\n\n\n\n- In my case, I originally had process ID as a label. This is both not very useful and extremely high cardinality (the same service may spawn repeatedly for example) and ended up consuming a lot of inode space very quickly.<\/li>\n\n\n\n
- I added a
fix_cardinality<\/code>loki.process<\/code> stage in my alloy config where I added theproc_id <\/code>label as structured metadata and dropped the label before forwarding it to loki for write. This eliminated the high cardinality label and associated problems in my setup. Here’s the stage config in alloy for removing this high cardinality label:<\/li>\n<\/ul>\n\n\n\n\/\/ We pipe logs here BEFORE sending them to Loki.\nloki.process \"fix_cardinality\" {\n \/\/ Forward to the writer after processing\n forward_to = [loki.write.default.receiver]\n\n \/\/ 1: Move high-cardinality labels to Structured Metadata\n \/\/ This keeps \"proc_id\" searchable (e.g. | proc_id=\"1234\") but not indexed.\n stage.structured_metadata {\n values = {\n proc_id = \"\", \/\/ Automatically takes the value of the \"proc_id\" label\n }\n }\n\n \/\/ 3: Drop high cardinality proc_id label\n stage.label_drop {\n values = [\"proc_id\"]\n }\n}<\/code><\/pre>\n\n\n\n- \n
- Loki will automatically add a service_name<\/a> label to ingested log lines that is discovered based on a default set of rules.<\/a> In my case, this label is redundant since
rsyslog<\/code> will forward entries withapp_name<\/code> set appropriately andservice_name<\/code> is just a repeat of the same info. So I disabled it by adding this to my loki config.ymllimits_config<\/code>:discover_service_name: []<\/code><\/li>\n\n\n\n- Loki’s ingester defaults are not meant to be tuned for a typical home network environment where most nodes produce sporadic and low volume of logs. You can adjust the settings to reduce the amount of files created by loki by tuning these sections – the first parameter dictates the maximum amount of time a chunk is held open and the second parameter dictates how long a chunk can be kept open even if no new logs were received for it:<\/li>\n<\/ul>\n\n\n\n
ingester:\n max_chunk_age: 24h\n chunk_idle_period: 12h<\/code><\/pre>\n\n\n\n<\/p>\n","protected":false},"excerpt":{"rendered":"
Finally got some time to put all the pieces together for collecting logs from across my Home Network and centralizing them for analysis, visualization. The general setup is the following: In my case, I chose the following setup: Here’s a visual diagram of the system with various components and how they connect to each other….<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-609","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts\/609","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/comments?post=609"}],"version-history":[{"count":3,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts\/609\/revisions"}],"predecessor-version":[{"id":615,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts\/609\/revisions\/615"}],"wp:attachment":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/media?parent=609"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/categories?post=609"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/tags?post=609"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Loki’s ingester defaults are not meant to be tuned for a typical home network environment where most nodes produce sporadic and low volume of logs. You can adjust the settings to reduce the amount of files created by loki by tuning these sections – the first parameter dictates the maximum amount of time a chunk is held open and the second parameter dictates how long a chunk can be kept open even if no new logs were received for it:<\/li>\n<\/ul>\n\n\n\n
- You can also check how much disk space you are using and how much inode space you are using (note that you may run out of inode space even though you have plenty of disk space) using the
- The central log server (also running rsyslog and receiving logs from all nodes), saves them to files and forwards them to an alloy<\/a> host. The rsyslog server processes the logs to add necessary fields that are helpful in queries – you can read more about how this was setup here.<\/a><\/li>\n\n\n\n