Web Portal and Collectd/rrd graphs automated installation

[mgunther]

Thanks bramj, that works! I actually removed the sizing parameters from /var/www/collectd/index.html because they are not needed when the png images are already at the desired size. The section you mentioned looks like this now:

    <div style="width: 2390px; height: 273px;">
        <img id="rate" src="dump1090-rpi-rate-24h.png">
        <img id="acs" src="dump1090-rpi-acs-24h.png">
    </div>
    <div style="width: 2390px; height: 273px;">   
        <img id="cpu" src="dump1090-rpi-cpu-24h.png">
        <img id="tracks" src="dump1090-rpi-tracks-24h.png">
    </div>
    <div style="width: 2390px; height: 273px;">   
        <img id="range" src="dump1090-rpi-range-24h.png">
        <img id="ctemp" src="table-rpi-core_temp-24h.png">
    </div>

The temperature graph was not as high as the other ones because it had no legend. I fixed this in /home/pi/dump-tools/collectd/make-graphs.sh by adding ":temperature" here:

metric_temp_graph() {
  rrdtool graph \
  "$1" \
  --start end-$4 \
  --width 480 \
  --height 200 \
  --step "$5" \
  --title "Core Temperature" \
  --vertical-label "Degrees Celcius" \
  --lower-limit 0 \
  --upper-limit 100 \
  --rigid \
  --units-exponent 1 \
  "DEF:traw=$2/gauge-cpu_temp.rrd:value:MAX" \
  "CDEF:tfin=traw,1000,/" \
  "AREA:tfin#ffcc00:temperature"
}

 

[mgunther]

your width and height in the .sh script don't seem to exactly match your html image settings/values but that would result in some slightly streched graphs (possibly)

Yes, in my opinion, it's not a good idea to generate image at one size and then stretch/squash them to fit another size. Best to generate them exactly at the desired size and then display them without distortion

 

[jepolch]

I've been playing with some of the features of RRD tool to produce some extra information:

I added a trailing 7 day average. This is the mean average of the preceding 7 days message rate, not including the current day. The yellow area is the range of the data over those 7 days. This is pretty useful for comparing a change to receiver configuration - you can immediately see if the current days data falls within the previously established norms. This obviously works best if you make a change after a stable period of at least 7 days, and you will see greater variance if you make a lot of changes in quick succession.

I modified obj's script to produce this graph. This is the relevant section of the code:

local_rate_graph() {
  rrdtool graph \
  "$1" \
  --start end-$4 \
  --width 823 \
  --height 217 \
  --step "$5" \
  --title "$3 message rate" \
  --vertical-label "messages/second" \
  --lower-limit 0  \
  --units-exponent 0 \
  --right-axis 360:0 \
  --slope-mode \
  "DEF:messages=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE" \

#Get the average value for each of the preceeding 7 days.

  "DEF:a=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-86400:start=end-86400" \
  "DEF:b=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-172800:start=end-86400" \
  "DEF:c=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-259200:start=end-86400" \
  "DEF:d=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-345600:start=end-86400" \
  "DEF:e=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-432000:start=end-86400" \
  "DEF:f=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-518400:start=end-86400" \
  "DEF:g=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-604800:start=end-86400" \

#Gets the minimum values for the preceeding 7 days.

  "DEF:amin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-86400:start=end-86400" \
  "DEF:bmin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-172800:start=end-86400" \
  "DEF:cmin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-259200:start=end-86400" \
  "DEF:dmin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-345600:start=end-86400" \
  "DEF:emin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-432000:start=end-86400" \
  "DEF:fmin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-518400:start=end-86400" \
  "DEF:gmin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-604800:start=end-86400" \

#Get the maximum value for the preceeding 7 days

  "DEF:amax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-86400:start=end-86400" \
  "DEF:bmax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-172800:start=end-86400" \
  "DEF:cmax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-259200:start=end-86400" \
  "DEF:dmax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-345600:start=end-86400" \
  "DEF:emax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-432000:start=end-86400" \
  "DEF:fmax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-518400:start=end-86400" \
  "DEF:gmax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-604800:start=end-86400" \

  "DEF:strong=$2/dump1090_messages-strong_signals.rrd:value:AVERAGE" \
  "DEF:positions=$2/dump1090_messages-positions.rrd:value:AVERAGE" \
  "CDEF:y2strong=strong,10,*" \
  "CDEF:y2positions=positions,10,*" \

#Shift the historical data in time so it can be overlaid with the current data.

  "SHIFT:a:86400" \
  "SHIFT:b:172800" \
  "SHIFT:c:259200" \
  "SHIFT:d:345600" \
  "SHIFT:e:432000" \
  "SHIFT:f:518400" \
  "SHIFT:g:604800" \
  "SHIFT:amin:86400" \
  "SHIFT:bmin:172800" \
  "SHIFT:cmin:259200" \
  "SHIFT:dmin:345600" \
  "SHIFT:emin:432000" \
  "SHIFT:fmin:518400" \
  "SHIFT:gmin:604800" \
  "SHIFT:amax:86400" \
  "SHIFT:bmax:172800" \
  "SHIFT:cmax:259200" \
  "SHIFT:dmax:345600" \
  "SHIFT:emax:432000" \
  "SHIFT:fmax:518400" \
  "SHIFT:gmax:604800" \

#Calculate the 7 day average

  "CDEF:7dayaverage=a,b,c,d,e,f,g,+,+,+,+,+,+,7,/" \

#Find the minimum and maximum data from the preceeding 7 days. This could probably be done in two lines, but I'm not a programmer and RPN makes my head hurt.

  "CDEF:min1=amin,bmin,MINNAN" \
  "CDEF:min2=cmin,dmin,MINNAN" \
  "CDEF:min3=emin,fmin,MINNAN" \
  "CDEF:min4=min1,min2,MINNAN" \
  "CDEF:min5=min3,gmin,MINNAN" \
  "CDEF:min=min4,min5,MINNAN" \
  "CDEF:max1=amax,bmax,MAXNAN" \
  "CDEF:max2=cmax,dmax,MAXNAN" \
  "CDEF:max3=emax,fmax,MAXNAN" \
  "CDEF:max4=max1,max2,MAXNAN" \
  "CDEF:max5=max3,gmax,MAXNAN" \
  "CDEF:max=max4,max5,MAXNAN" \

#Calculation in order to get an area between max and min values.

  "CDEF:maxarea=max,min,-" \

#Draw graphs.

  "LINE1:min#FFFF99:mins" \
  "AREA:maxarea#FFFF99:max:STACK" \
  "LINE1:7dayaverage#00FF00:7 day average" \
  "AREA:y2strong#FF0000:messages >-3dBFS / hr (RHS)" \
  "LINE1:y2positions#00c0FF:positions / hr (RHS)" \
  "LINE1:messages#0000FF:messages received"
}

If you just modify the existing script with this, then you will get the overlaid bits for a short section on the week, month and year graphs as well. It looks a bit odd, so you might want to create a separate script to draw this one.

I also found in the RRD tool documentation that it is possible to make predictions based on past data. This graph shows 7 days of real data, and a prediction for 3 days ahead. It also shows a confidence interval of +/- 2 standard deviations. The prediction is based on the last 14 days of data:

You can see the confidence interval starts wide and gets closer - that is because the first few days are still using some data from before I switched receivers to the R820T2, which has better performance than the old one. The highlighted areas are where the data falls outside the confidence interval.

The script that produced this graph is modified from an example in the RRD tool documentation:

#!/bin/sh

tracks_graph() {
  rrdtool graph \
  /var/www/collectd/predict.png \
  --start=-7days \
  --end=+3days \
  --width=1800 \
  --height=850 \
  --alt-autoscale-max \
  --slope-mode \
  --title="Message rate forecast" \
  "DEF:value=/var/lib/collectd/rrd/localhost/dump1090-dump1090/dump1090_messages-local_accepted.rrd:value:AVERAGE:start=-14days" \
  "LINE1:value#ff0000:value" \
  "CDEF:predict=86400,-7,1800,value,PREDICT" \
  "CDEF:sigma=86400,-7,1800,value,PREDICTSIGMA" \
  "CDEF:upper=predict,sigma,2,*,+" \
  "CDEF:lower=predict,sigma,2,*,-" \
  "LINE1:predict#00ff00:prediction" \
  "LINE1:upper#0000ff:upper certainty limit" \
  "LINE1:lower#0000ff:lower certainty limit" \
  "CDEF:exceeds=value,UN,0,value,lower,upper,LIMIT,UN,IF" \
  "TICK:exceeds#F6D8CE80:1"
}

tracks_graph

I don't know if it's of any practical use, but it's there if anyone wants to play with it.

Very, very nice work!

 

[jepolch]

@xforce30164 et al: It would be easier for me to compare my two receivers if the numbers on the left axis were the same ... I'm not really sure how to describe what I'm saying, so I'll show a couple of examples. If the numbers on the left axis were fixed rather than variable, it would be easier to compare the two receivers. Know what I mean?

Is there any way to do that?

 

[caius]

You can do that - add the following parameters to the script for the graph you want to change:

  --upper-limit 100  \
  --lower-limit 0  \

Set the numbers to what ever you want. You will see that some graphs like the signal strength one already have a fixed vertical axis, so you can see where they are supposed to go.

 

[bramj]

That was exactly what I was going to suggest! Thanks @bramj for the quick response =]. (though your width and height in the .sh script don't seem to exactly match your html image settings/values but that would result in some slightly streched graphs (possibly)
and another thing to put on the dynamic script configurator, image size setting

,
"Note the width and height define the graph size and not the size of the entire picture"

 

[xforce30164]

,
"Note the width and height define the graph size and not the size of the entire picture"

Aaah! I didn't know that, learned something new again today =D

 

[xforce30164]

@xforce30164 et al: It would be easier for me to compare my two receivers if the numbers on the left axis were the same ... I'm not really sure how to describe what I'm saying, so I'll show a couple of examples. If the numbers on the left axis were fixed rather than variable, it would be easier to compare the two receivers. Know what I mean?

Is there any way to do that?

Do as caius said. also i might rerun my own installtion script as you also seem to get the trailing acerage graph to work whilst i cant.

 

[jepolch]

You can do that - add the following parameters to the script for the graph you want to change:

  --upper-limit 100  \
  --lower-limit 0  \

Set the numbers to what ever you want. You will see that some graphs like the signal strength one already have a fixed vertical axis, so you can see where they are supposed to go.

That's exactly what I wanted. Thanks!

 

[ab cd]

@xforce30164:

I have noted that when dump1090-mut dies, and I restart it, "collectd" does not resume its function. The graph plotting which has stopped with death of dump1090-mut does not get restarted unless i use "sudo service collected restart".

Piaware & FR24feed behave differently. These immediately sense the restart of dump1090-mut, and resume data feeding without manual restart.

 

[xforce30164]

@xforce30164:

I have noted that when dump1090-mut dies, and I restart it, "collectd" does not resume its function. The graph plotting which has stopped with death of dump1090-mut does not get restarted unless i use "sudo service collected restart".

Piaware & FR24feed behave differently. These immediately sense the restart of dump1090-mut, and resume data feeding without manual restart.

I'll put it on my to-do list and see If i can look into this/make something similar. (Maybe run a script from crontab that checks if collectd is running, and if not (restarts) it.)

 

[xforce30164]

Post by bramj in the other thread on how to add the signal graph. the reason I initially didn't add it was because it didn't work for me but that was due to the fact that I was also deocoding mode-AC which disables oversampling (which is needed to generate the signal strength data)

Add the bold line ...to the make script

common_graphs(){
aircraft_graph /var/www/collectd/dump1090-$2-acs-$4.png /var/lib/collectd/rrd/$1/dump1090-$2 "$3" "$4" "$5"
cpu_graph /var/www/collectd/dump1090-$2-cpu-$4.png /var/lib/collectd/rrd/$1/dump1090-$2 "$3" "$4" "$5"
tracks_graph /var/www/collectd/dump1090-$2-tracks-$4.png /var/lib/collectd/rrd/$1/dump1090-$2 "$3" "$4" "$5"
metric_temp_graph /var/www/collectd/table-$2-core_temp-$4.png /var/lib/collectd/rrd/$1/table-$2 "$3" "$4" "$5"
#imperial_temp_graph /var/www/collectd/table-$2-core_temp-$4.png /var/lib/collectd/rrd/$1/table-$2 "$3" "$4" "$5"
metric_range_graph /var/www/collectd/dump1090-$2-range-$4.png /var/lib/collectd/rrd/$1/dump1090-$2 "$3" "$4" "$5"
#imperial_range_graph /var/www/collectd/dump1090-$2-range-$4.png /var/lib/collectd/rrd/$1/dump1090-$2 "$3" "$4" "$5"

signal_graph /var/www/collectd/dump1090-$2-signal-$4.png /var/lib/collectd/rrd/$1/dump1090-$2 "$3" "$4" "$5"

}

start the script sudo ./make-graph 1h
This creates dump1090-rpi-signal-1h.png in /var/www/collectd/

start the script sudo ./make-graph 24h
This creates dump1090-rpi-signal-24h.png in /var/www/collectd/


edit the index.html in /var/www/collectd

function switchView(time_name){

$("#rate").attr("src", base_name + "-rate-" + time_name + ".png");

$("#acs").attr("src", base_name + "-acs-" + time_name + ".png");

$("#cpu").attr("src", base_name + "-cpu-" + time_name + ".png");

$("#tracks").attr("src", base_name + "-tracks-" + time_name + ".png");

$("#range").attr("src", base_name + "-range-" + time_name + ".png");

$("#ctemp").attr("src", "table-rpi-core_temp-" + time_name + ".png");

$("#signal").attr("src", base_name + "-signal-" + time_name + ".png");
}

.
.
.

<div style="width: 1280px; height: 273px;">

<img id="rate" src="dump1090-rpi-rate-24h.png">

<img id="acs" src="dump1090-rpi-acs-24h.png">

</div>

<div style="width: 1280px; height: 273px;">

<img id="cpu" src="dump1090-rpi-cpu-24h.png">

<img id="tracks" src="dump1090-rpi-tracks-24h.png">

</div>

<div style="width: 1280px; height: 273px;">

<img id="range" src="dump1090-rpi-range-24h.png">

<img id="ctemp" src="table-rpi-core_temp-24h.png">

</div>

<div style="width: 1280px; height: 273px;">

<img id="signal" src="dump1090-rpi-signal-24h.png">

</div>


View attachment 1737

 

[bramj]

I added 1 hour and 6 hour graph

edit in /var/www/collectd/ the index.html

<html>
    <head>
        <meta http-equiv="refresh" content="60">
        <title> Collectd Dump1090 Graphs</title>
        <script type="text/javascript" src="jquery.js"></script>
        <script type="text/javascript">
            var base_name = "dump1090-rpi";
     
            function switchView(time_name){
                $("#rate").attr("src", base_name + "-rate-" + time_name + ".png");
                $("#acs").attr("src", base_name + "-acs-" + time_name + ".png");
                $("#cpu").attr("src", base_name + "-cpu-" + time_name + ".png");
                $("#tracks").attr("src", base_name + "-tracks-" + time_name + ".png");
                $("#range").attr("src", base_name + "-range-" + time_name + ".png");
                $("#ctemp").attr("src", "table-rpi-core_temp-" + time_name + ".png");
                $("#signal").attr("src", base_name + "-signal-" + time_name +  ".png");         
}
        </script>
    </head>
 
    <body>
    <p>
     <a href="#" onclick="switchView('1h')">[1-hour ]</a> <a href="#" onclick="switchView('6h')">[ 6-hour ]</a> <a href="#" onclick="switchView('24h')">[ day ]</a> <a href="#" onclick="switchView('7d')">[ week ]</a> <a href="#" onclick="switchView('30d')">[ month ]</a> <a href="#" onclick="switchView('365d')">[ year ]</a>
    </p>

    <div style="width: 1280px; height: 273px;">
        <img id="rate" src="dump1090-rpi-rate-24h.png">
        <img id="cpu" src="dump1090-rpi-cpu-24h.png">

    </div>
    <div style="width: 1280px; height: 273px;">
<img id="acs" src="dump1090-rpi-acs-24h.png">    
        <img id="tracks" src="dump1090-rpi-tracks-24h.png">
    </div>
    <div style="width: 1280px; height: 273px;">
        <img id="range" src="dump1090-rpi-range-24h.png">
        <img id="ctemp" src="table-rpi-core_temp-24h.png">
    </div>
   <div style="width: 1280px; height: 273px;">
        <img id="signal" src="dump1090-rpi-signal-24h.png">
    </div>

    </body>
</html>
[B]

[/B]



edit crontab with crontab -e

*/5 * * * * sudo /home/pi/dump-tools/collectd/make-graphs.sh 1h >/dev/null
*/10 * * * * sudo /home/pi/dump-tools/collectd/make-graphs.sh 6h >/dev/null
2,12,22,32,42,52 * * * * sudo /home/pi/dump-tools/collectd/make-graphs.sh 24h >/dev/null
4,24,44 * * * * sudo /home/pi/dump-tools/collectd/make-graphs.sh 7d >/dev/null
6 * * * * sudo /home/pi/dump-tools/collectd/make-graphs.sh 30d >/dev/null
8 */12 * * * sudo /home/pi/dump-tools/collectd/make-graphs.sh 365d >/dev/null

if you don't want to wait until cron start the make-graphs.sh script

cd ~/dump-tools/collectd/
sudo ./make-graphs.sh 1h
sudo ./make-graphs.sh 6h

 

[loplo]

If someone is going to edit files in windows, and has issues running the scripts, should install dos2unix and convert the files (after editing) using:

sudo dos2unix path_to_filename

 

[xforce30164]

I'm working on an improved installation script to make it much easier to push small updates for any new programs/tools im writing:
here's a sneak preview screenshot:

 

[caius]

If someone is going to edit files in windows, and has issues running the scripts, should install dos2unix and convert the files (after editing) using:

sudo dos2unix path_to_filename

Alternatively, you could use an editor that can write unix text files natively - I use Notepad++ which has a lot of nice features including colour markup for most common file formats. It also has a lot of plugins available to add features.

 

[xforce30164]

Alternatively, you could use an editor that can write unix text files natively - I use Notepad++ which has a lot of nice features including colour markup for most common file formats. It also has a lot of plugins available to add features.

I also use notepad++ a lot, really clean ui but lots of useful possibilities!

 

[loplo]

Notepad++ is one of the first pieces of software I install after a fresh OS installation. Although, it happens, depending of the source of the original files, that the files are going to be saved in dos/windows format.

 

[caius]

It does tell you at right of the status bar at the bottom which EOL format is in use. You can change it easily by selecting EOL conversion from the edit menu.

 

[loplo]

I know. I posted the info for those not knowing what's happening, or not using N++