How to install and configure fluent-bit on macOS

Fluent Bit is a Fast and Lightweight Logs and Metrics Processor and Forwarder for Linux, OSX, Windows and BSD family operating systems. It has been made with a strong focus on performance to allow the collection of events from different sources without complexity.

In this article, we’ll get started with fluent-bit, part 1 with installing it on macOS and configuring it to tail an Apache log file.

Prerequisite:

  • Homebrew

Install fluent-bit:

  • let’s install the fluent-bit on macOS using Homebrew:
brew install fluent-bit
  • Once it's installed, the binary will be in location : /usr/local/bin/fluent-bit
  • You can test if the binary is working correctly or not by running : fluent-bit -V which should display Fluent Bit v1.9.3 or similar version.

Install flog:

  • Flog is a fake log generator for common log formats such as apache-common, apache error and RFC3164 syslog.
  • Download the binary from GitHub :
wget https://github.com/mingrammer/flog/releases/download/v0.4.3/flog_0.4.3_darwin_amd64.tar.gz
  • Extract the binary using tar :
tar -xzvf flog_0.4.3_darwin_amd64.tar.gz
  • Run the flog to dump a fake Apache logs into a file :
./flog > /tmp/flog.log

Configure fluent-bit :

Now that the log file is ready and fluent-bit installed, we’ll create a file called fluent-bit.conf and append the below files

[SERVICE]
log_level info
flush 1
[INPUT]
Name tail
path /tmp/flog.log
Read_from_Head true
[OUTPUT]
Name stdout
Match *
  • Let’s take a look at the config file to understand what it's doing :
  1. SERVICE: This is fluent-bit's config section.
  • flush: this means on what time interval (seconds) it should push the logs to output. Here it's 1 i.e. every second.

2. INPUT: This defines what to collect and where to collect it from.

  • Name: the name of the input plugin. The tail input plugin allows to monitor one or several text files. Check more on input plugin here.
  • path: specifies path of file we want to read and its name.
  • Read_from_Head: specifies to read the file from starting. This is necessary of the file you're reading has already contents in it and you want to grab those too. Else, anything else added to the file after fluent-bit starts will only be taken into consideration.

3. OUTPUT: defines where to send the logs it read in INPUT section.

  • Name: the name of the output plugin. In this case we're not sending the logs to any ElasticSearch or Loki etc. We're just routing the input to stdout. You can read more about stdout plugin here.
  • match: Using tags, we have the ability to send certain section of logs only to outputs. Since we’re not mentioning any tags in previous section, we’re saying send everything to stdout with * .

Action time

  • Let’s start the fluent-bit with the config-file we created :
./fluent-bit -c fluent-bit.conf
  • Your output should now be similar to :
Fluent Bit v1.9.3* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/05/28 21:21:10] [ info] [output:stdout:stdout.0] worker #0 started
[2022/05/28 21:21:10] [ info] [sp] stream processor started
[1993] tail.0: [1653753034.293609000, {"log"=>"33.28.73.181 - gaylord8047 [28/May/2022:20:51:41 +0530] "GET /robust/matrix HTTP/1.0" 416 21018"}]
  • There you go, fluent-bit working smooth as butter.
  • Keep in mind, we haven’t done any filtering or parsing on the log files. We’ll do that in the upcoming part of the series.

--

--

--

DevOps @ unbxd.com || tanmay-bhat.github.io

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Importance of teamwork

5 Minute Hacks: Simple Automations To Help You Stay Organised

Preparing for a Technical Assessment that Requires Data Merging Tasks in Python

Path To DevFolio, Nav Widget Wiring

Load data from PostgreSQL into Autonomous Database using Oracle GoldenGate

Extreme Programming (Part Two): Test-Driven Development and Small-Batch Development

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tanmay Bhat

Tanmay Bhat

DevOps @ unbxd.com || tanmay-bhat.github.io

More from Medium

Parsing Apache & Nginx logs with fluent-bit

Nginx Ingress Controller Installation On k8s Cluster

Deploying EFK stack on Kubernetes

Monitor Your Computing System with Prometheus, Grafana, Alertmanager, and Nvidia DCGM