Exporting custom metrics to influxdb

In the TICK stack (Telegraf, InfluxDB, Chronograf, Kapacitor) from InfluxData, Telegraf serves as the crucial "T"—the agent responsible for collecting and sending data. Metrics stored in InfluxDB can then be transformed into actionable insights using visualization tools like Grafana.

1. Configuring Telegraf Plugins

Telegraf's power lies in its pluggable architecture. By modifying the /etc/telegraf/telegraf.conf file, you can enable a wide range of input and output plugins to suit your infrastructure.

Enabling Input and Output Plugins

For example, to export swap information from a host with a custom tag, you would add the following to your configuration:

[[inputs.swap]]   [inputs.swap.tags]     metrics_source="telegraf_demo" 

To verify the data flow during development, you can use the file output plugin to pipe metrics directly to stdout or a local file in InfluxDB line protocol format:

[[outputs.file]]   files = ["stdout", "/tmp/metrics.out"]   data_format = "influx" 

2. Containerizing the Telegraf Agent

When monitoring containerized environments, it is common practice to run Telegraf itself within a container.

Building and Running the Telegraf Image

We can create a simple Dockerfile to bundle our custom configuration:

FROM telegraf COPY telegraf.conf /etc/telegraf/telegraf.conf CMD ["telegraf"] 

Build the container: # docker build -t telegrafdemo .

When running the container, you will see the output plugin in action, displaying metrics formatted as InfluxDB Line Protocol:

swap,metrics_source=telegraf_demo,host=e9e7086036b8 total=8271163392i,used=0i,free=8271163392i 1514876555000000000 

3. Extending Functionality with the Exec Plugin

If the built-in plugins don't meet your specific needs, the Exec Plugin allows you to execute external scripts and collect their output.

Creating a Custom Collection Script

The following script parses the output of the uptime command and formats it for InfluxDB:

#!/sh hostname=`hostname` uptime=`awk '{print $1}' /proc/uptime` # ... (logic to extract load averages) echo "uptime,host=$hostname uptime=$uptime,load1=$load1,load5=$load5,load15=$load15" 

Running the script manually confirms the output format: # sh uptime.sh uptime,host=localhost.localdomain uptime=25524.65,load1=1.14,load5=1.06,load15=1.04

4. Full Stack Demonstration: Connecting to InfluxDB

To see the end-to-end flow, we need to connect Telegraf to a running InfluxDB instance using a Docker network.

Setting Up the Network and InfluxDB

# Create a dedicated bridge network docker network create --driver=bridge mintonw  # Start InfluxDB container docker run --net=mintonw --name influxdemo influxdb 

Final Telegraf Configuration

Update your telegraf.conf to point to the InfluxDB container and include the custom script:

[[inputs.exec]]   commands = ["/etc/telegraf/uptime.sh"]   data_format = "influx"  [[outputs.influxdb]]   url = "http://influxdemo:8086"   database = "telegraf" 

Verifying the Data in InfluxDB

Once the Telegraf container is running on the same network, you can query InfluxDB to verify the custom metrics:

> use telegraf > show measurements name: measurements ---- swap uptime  > select * from uptime limit 3 time                host          load1  load15  load5  metrics_source  uptime ----                ----          -----  ------  -----  --------------  ------ 1514874715000000000 3595f01eba01  0.26   0.66    0.54   telegraf_demo   13198.45 

Custom metrics can be exported easily using Telegraf's exec plugin. While this example uses uptime for demonstration, remember that many standard system metrics can be collected using native Telegraf plugins if the host's /proc or /var/run directories are mounted into the container.