Prometheus node_exporter with perccli textfile collector on xenserver

If using prometheus/grafana setup for monitoring you probably used node_exporter for exporting machine metrics.

I will show you how to install node_exporter together with perccli utility on xenserver host. Prior to node_exporter 0.15.0 megacli was integrated as collector (only disabled by default) and used megacli utility. But with version >0.15.0 it is no longer integrated and if you wanna monitor your RAID status you need to use that utility as textfile collector.

node_exporter

To install node_exporter just download it from: https://github.com/prometheus/node_exporter/releases. Get latest version 0.15.2. Do it directly on xenserver host or copy it after:

wget https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz -P /tmp

Export it and move it to /opt directory for example.

tar xvf node_exporter-0.15.2.linux-amd64.tar.gz
mv node_exporter-0.15.2.linux-amd64 /opt/node_exporter

By default node_exporter is listening on port 9100 so for prometheus to scrape it you need to allow port 9100 on xenserver iptables:

sed -i -e '/RH-Firewall-1-INPUT -j REJECT/i\-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 9100 -j ACCEPT' /etc/sysconfig/iptables && service iptables restart

Then create systemd service for node_exporter and start it.

cat /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter

[Service]
User=root
ExecStart=/opt/node_exporter/node_exporter --collector.systemd 

[Install]
WantedBy=default.target

Nothing special in this service file except I usually enable systemd collector to be able to see service and system status from systemd. After creating service file restart deamon, enable and start node_exporter.service

systemctl daemon-reload
systemctl enable node_exporter.service
systemctl start node_exporter.service

After this you should be able to see metrics going to http://your-server-ip:9100/metrics

perccli

If you have LSI SAS card you should use storecli utility ^[1] (it replaces megacli) but since some of my servers are DELL R710 with PERC H700 controller storcli is unable to check the controller so I need to use perccli utility ^[2].

After downloading utility to xenserver, extract it and install the package

rpm -ivh /tmp/perccli-1.11.03-1.noarch.rpm

Utility will, by default, install in /opt/MegaRAID/perccli/. Now to use utility with node_exporter you need script that will run perccli utility and format output so prometheus is able to use it. There is a python script on node_exporter github https://github.com/prometheus/node_exporter/tree/master/text_collector_examples named storecli.py. So if you are using storecli you can use this one, but if you are stuck with perccli there is small change you need to make in this script. At the end just change path from storecli to perccli:

---
PARSER.add_argument('--storcli_path',
                        default='/opt/MegaRAID/perccli/perccli64',
---

Now you need to create cron job that will start script and get output into directory which you will also specify in node_exporter service.

mkdir -p /var/lib/node_exporter/textfile_collector

And put into root crontab:

*/10 * * * * /usr/bin/python /opt/node_exporter/perccli.py > /var/lib/node_exporter/textfile_collector/perccli.prom.$$ && mv /var/lib/node_exporter/textfile_collector/perccli.prom.$$ /var/lib/node_exporter/textfile_collector/perccli.prom

As you can see I put perccli.py script into /opt/node_exporter and script output file into /var/lib/node_exporter/textfile_collector directory, so if you need then adjust paths accordingly. I also run this job every 10min.

Now you need to push this into node_exporter and that is done by edit node_exporter.service file you created earlier:

[Unit]
Description=Node Exporter

[Service]
User=root
ExecStart=/opt/node_exporter/node_exporter --collector.systemd --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

[Install]
WantedBy=default.target

After that reload daemon, and restart service:

systemctl daemon-reload
service node_exporter restart

Now if you go again to http://your-server-ip:9100/metrics you can see megaraid info among others

That's it. Just add target to prometheus and create your queries.