Web Portal and Collectd/rrd graphs automated installation

ab cd

ab cd

Senior Member
Yesteday I ran @xforce30164's installer script on both of my RPis. One RPi had collected installed (but not working), while other RPi didn't have rrdtool or collectd installed.

The script installed successfully on both RPis, with graphs installed and running.

Great work @xforce30164! Thank you.

21 hrs after installation

Screenshot_2015-04-19-00-21-17.png
 
xforce30164

xforce30164

Active Member
bramj said:
maybe i missed something,
What webserver package is used Apache , lighttp , nginx ...?

Bram
Click to expand...

I'm using the lighttpd server package. (this is what was advised by obj himself), but I'll add this to the description/starting post

so run:
Code: 
sudo apt-get install lighttpd && sudo lighty-enable-mod dump1090
and follow any additional tips/commands from lighttpd installation.
(You'll also have to edit your dump1090-muta config to disable internal webserver. IIRC this means changing the web-port to 0 (instead of 8080 or something similar))
 
jepolch

jepolch

Active Member
@xforce30164: Congratulations on your new blog! It looks very nice. I know it's a work in progress and will be evolving as you make changes to your scripts. There are a couple of things that could use a little more explanation. Why would I want to update or revert? Is this to get newer versions of make-graphs.sh and the other scripts?
 
bramj

bramj

Member
xforce30164 said:
I'm using the lighttpd server package. (this is what was advised by obj himself), but I'll add this to the description/starting post

so run:
Code: 
sudo apt-get install lighttpd && sudo lighty-enable-mod dump1090
and follow any additional tips/commands from lighttpd installation.
(You'll also have to edit your dump1090-muta config to disable internal webserver. IIRC this means changing the web-port to 0 (instead of 8080 or something similar))
Click to expand...

Aha , that explains it :)
Screen Shot 2015-04-19 at 21.06.22.png
Screen Shot 2015-04-19 at 21.06.32.png
 
xforce30164

xforce30164

Active Member
jepolch said:
@xforce30164: Congratulations on your new blog! It looks very nice. I know it's a work in progress and will be evolving as you make changes to your scripts. There are a couple of things that could use a little more explanation. Why would I want to update or revert? Is this to get newer versions of make-graphs.sh and the other scripts?
Click to expand...
jup, as I will be developing Ill be probably adding more functions etc. then you can grab those with update. with revert you can restore the previous configuration file so if you messed up when addinf your own stuff you can easily get it back to a working state and try again
 
caius

caius

Member
I've been playing with some of the features of RRD tool to produce some extra information:

E7BA4qw.png


I added a trailing 7 day average. This is the mean average of the preceding 7 days message rate, not including the current day. The yellow area is the range of the data over those 7 days. This is pretty useful for comparing a change to receiver configuration - you can immediately see if the current days data falls within the previously established norms. This obviously works best if you make a change after a stable period of at least 7 days, and you will see greater variance if you make a lot of changes in quick succession.

I modified obj's script to produce this graph. This is the relevant section of the code:

Code: 
local_rate_graph() {
  rrdtool graph \
  "$1" \
  --start end-$4 \
  --width 823 \
  --height 217 \
  --step "$5" \
  --title "$3 message rate" \
  --vertical-label "messages/second" \
  --lower-limit 0  \
  --units-exponent 0 \
  --right-axis 360:0 \
  --slope-mode \
  "DEF:messages=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE" \

#Get the average value for each of the preceeding 7 days.

  "DEF:a=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-86400:start=end-86400" \
  "DEF:b=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-172800:start=end-86400" \
  "DEF:c=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-259200:start=end-86400" \
  "DEF:d=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-345600:start=end-86400" \
  "DEF:e=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-432000:start=end-86400" \
  "DEF:f=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-518400:start=end-86400" \
  "DEF:g=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-604800:start=end-86400" \

#Gets the minimum values for the preceeding 7 days.

  "DEF:amin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-86400:start=end-86400" \
  "DEF:bmin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-172800:start=end-86400" \
  "DEF:cmin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-259200:start=end-86400" \
  "DEF:dmin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-345600:start=end-86400" \
  "DEF:emin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-432000:start=end-86400" \
  "DEF:fmin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-518400:start=end-86400" \
  "DEF:gmin=$2/dump1090_messages-local_accepted.rrd:value:MIN:end=now-604800:start=end-86400" \

#Get the maximum value for the preceeding 7 days

  "DEF:amax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-86400:start=end-86400" \
  "DEF:bmax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-172800:start=end-86400" \
  "DEF:cmax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-259200:start=end-86400" \
  "DEF:dmax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-345600:start=end-86400" \
  "DEF:emax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-432000:start=end-86400" \
  "DEF:fmax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-518400:start=end-86400" \
  "DEF:gmax=$2/dump1090_messages-local_accepted.rrd:value:MAX:end=now-604800:start=end-86400" \

  "DEF:strong=$2/dump1090_messages-strong_signals.rrd:value:AVERAGE" \
  "DEF:positions=$2/dump1090_messages-positions.rrd:value:AVERAGE" \
  "CDEF:y2strong=strong,10,*" \
  "CDEF:y2positions=positions,10,*" \

#Shift the historical data in time so it can be overlaid with the current data.

  "SHIFT:a:86400" \
  "SHIFT:b:172800" \
  "SHIFT:c:259200" \
  "SHIFT:d:345600" \
  "SHIFT:e:432000" \
  "SHIFT:f:518400" \
  "SHIFT:g:604800" \
  "SHIFT:amin:86400" \
  "SHIFT:bmin:172800" \
  "SHIFT:cmin:259200" \
  "SHIFT:dmin:345600" \
  "SHIFT:emin:432000" \
  "SHIFT:fmin:518400" \
  "SHIFT:gmin:604800" \
  "SHIFT:amax:86400" \
  "SHIFT:bmax:172800" \
  "SHIFT:cmax:259200" \
  "SHIFT:dmax:345600" \
  "SHIFT:emax:432000" \
  "SHIFT:fmax:518400" \
  "SHIFT:gmax:604800" \

#Calculate the 7 day average

  "CDEF:7dayaverage=a,b,c,d,e,f,g,+,+,+,+,+,+,7,/" \

#Find the minimum and maximum data from the preceeding 7 days. This could probably be done in two lines, but I'm not a programmer and RPN makes my head hurt.

  "CDEF:min1=amin,bmin,MINNAN" \
  "CDEF:min2=cmin,dmin,MINNAN" \
  "CDEF:min3=emin,fmin,MINNAN" \
  "CDEF:min4=min1,min2,MINNAN" \
  "CDEF:min5=min3,gmin,MINNAN" \
  "CDEF:min=min4,min5,MINNAN" \
  "CDEF:max1=amax,bmax,MAXNAN" \
  "CDEF:max2=cmax,dmax,MAXNAN" \
  "CDEF:max3=emax,fmax,MAXNAN" \
  "CDEF:max4=max1,max2,MAXNAN" \
  "CDEF:max5=max3,gmax,MAXNAN" \
  "CDEF:max=max4,max5,MAXNAN" \

#Calculation in order to get an area between max and min values.

  "CDEF:maxarea=max,min,-" \

#Draw graphs.

  "LINE1:min#FFFF99:mins" \
  "AREA:maxarea#FFFF99:max:STACK" \
  "LINE1:7dayaverage#00FF00:7 day average" \
  "AREA:y2strong#FF0000:messages >-3dBFS / hr (RHS)" \
  "LINE1:y2positions#00c0FF:positions / hr (RHS)" \
  "LINE1:messages#0000FF:messages received"
}

If you just modify the existing script with this, then you will get the overlaid bits for a short section on the week, month and year graphs as well. It looks a bit odd, so you might want to create a separate script to draw this one.

I also found in the RRD tool documentation that it is possible to make predictions based on past data. This graph shows 7 days of real data, and a prediction for 3 days ahead. It also shows a confidence interval of +/- 2 standard deviations. The prediction is based on the last 14 days of data:

bUuIamh.png


You can see the confidence interval starts wide and gets closer - that is because the first few days are still using some data from before I switched receivers to the R820T2, which has better performance than the old one. The highlighted areas are where the data falls outside the confidence interval.

The script that produced this graph is modified from an example in the RRD tool documentation:

Code: 
#!/bin/sh

tracks_graph() {
  rrdtool graph \
  /var/www/collectd/predict.png \
  --start=-7days \
  --end=+3days \
  --width=1800 \
  --height=850 \
  --alt-autoscale-max \
  --slope-mode \
  --title="Message rate forecast" \
  "DEF:value=/var/lib/collectd/rrd/localhost/dump1090-dump1090/dump1090_messages-local_accepted.rrd:value:AVERAGE:start=-14days" \
  "LINE1:value#ff0000:value" \
  "CDEF:predict=86400,-7,1800,value,PREDICT" \
  "CDEF:sigma=86400,-7,1800,value,PREDICTSIGMA" \
  "CDEF:upper=predict,sigma,2,*,+" \
  "CDEF:lower=predict,sigma,2,*,-" \
  "LINE1:predict#00ff00:prediction" \
  "LINE1:upper#0000ff:upper certainty limit" \
  "LINE1:lower#0000ff:lower certainty limit" \
  "CDEF:exceeds=value,UN,0,value,lower,upper,LIMIT,UN,IF" \
  "TICK:exceeds#F6D8CE80:1"
}

tracks_graph

I don't know if it's of any practical use, but it's there if anyone wants to play with it.
 
Last edited:
xforce30164

xforce30164

Active Member
Those are some awesome graphs man :D. I'm planning on making a kind of configurator/webpage where you can select what graphs you want and then have it generate the make-graphs.sh script for you, I'll definately put these graphs in there as well, ofcourse crediting you for creating them ;).

Also, where do you live or what awesome antenna do you have? Your message rates are so high!

ETA:
you have
Code: 
"DEF:value=/var/lib/collectd/rrd/localhost/dump1090-dump1090/dump1090_messages-local_accepted.rrd:value:AVERAGE:start=-14days" \

in your edited tracks graph but shouldn't it be localhost/dump1090-rpi/dump1090_ ?

also when I run the script, for the trailing average part it gives me these errors:
Code: 
./make-graphs.sh: 59: ./make-graphs.sh: DEF:a=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-86400:start=end-86400: not found
./make-graphs.sh: 69: ./make-graphs.sh: DEF:amin=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-local_accepted.rrd:value:MIN:end=now-86400:start=end-86400: not found
./make-graphs.sh: 79: ./make-graphs.sh: DEF:amax=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-local_accepted.rrd:value:MAX:end=now-86400:start=end-86400: not found
./make-graphs.sh: 87: ./make-graphs.sh: DEF:strong=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-strong_signals.rrd:value:AVERAGE: not found
./make-graphs.sh: 94: ./make-graphs.sh: SHIFT:a:86400: not found
./make-graphs.sh: 118: ./make-graphs.sh: CDEF:7dayaverage=a,b,c,d,e,f,g,+,+,+,+,+,+,7,/: not found
./make-graphs.sh: 122: ./make-graphs.sh: CDEF:min1=amin,bmin,MINNAN: not found
./make-graphs.sh: 137: ./make-graphs.sh: CDEF:maxarea=max,min,-: not found
./make-graphs.sh: 141: ./make-graphs.sh: LINE1:min#FFFF99:mins: not found

even though the file paths seem to be ok.

prediction graph is working:
predict.png
 
Last edited:
mgunther

mgunther

New Member
Ran install script today. It worked without a hitch and graphs are starting to populate. Well done and thank you!
I wonder why the CPU graph is not left-aligned. Can you do anything about that?
 
caius

caius

Member
xforce30164 said:
Also, where do you live or what awesome antenna do you have? Your message rates are so high!

ETA:
you have
Code: 
"DEF:value=/var/lib/collectd/rrd/localhost/dump1090-dump1090/dump1090_messages-local_accepted.rrd:value:AVERAGE:start=-14days" \

in your edited tracks graph but shouldn't it be localhost/dump1090-rpi/dump1090_ ?

also when I run the script, for the trailing average part it gives me these errors:
Code: 
./make-graphs.sh: 59: ./make-graphs.sh: DEF:a=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-86400:start=end-86400: not found
./make-graphs.sh: 69: ./make-graphs.sh: DEF:amin=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-local_accepted.rrd:value:MIN:end=now-86400:start=end-86400: not found
./make-graphs.sh: 79: ./make-graphs.sh: DEF:amax=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-local_accepted.rrd:value:MAX:end=now-86400:start=end-86400: not found
./make-graphs.sh: 87: ./make-graphs.sh: DEF:strong=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-strong_signals.rrd:value:AVERAGE: not found
./make-graphs.sh: 94: ./make-graphs.sh: SHIFT:a:86400: not found
./make-graphs.sh: 118: ./make-graphs.sh: CDEF:7dayaverage=a,b,c,d,e,f,g,+,+,+,+,+,+,7,/: not found
./make-graphs.sh: 122: ./make-graphs.sh: CDEF:min1=amin,bmin,MINNAN: not found
./make-graphs.sh: 137: ./make-graphs.sh: CDEF:maxarea=max,min,-: not found
./make-graphs.sh: 141: ./make-graphs.sh: LINE1:min#FFFF99:mins: not found

even though the file paths seem to be ok.
Click to expand...

Message rate is mainly down to location. I live on the north east edge of London, so I can see all the traffic going in and out of Heathrow down to about 1000 feet. There is one of the approach holding stacks right above my house. I can also see most traffic for Stansted, Luton, and Gatwick. I can also see a band of high European traffic that flies along the coast from around Paris up to Amsterdam. Antenna is currently a spider with amp in the loft.

The path is down to me fiddling with the collectd config when I was setting it up. It works OK like that.

I don't know what's causing those errors. Perhaps there is a slight difference in the collectd config causing the saved data to be slightly different. I'll post my config later so you can compare.
 
xforce30164

xforce30164

Active Member
mgunther said:
Ran install script today. It worked without a hitch and graphs are starting to populate. Well done and thank you!
I wonder why the CPU graph is not left-aligned. Can you do anything about that?
Click to expand...
Not sure, you can shift graphs, but it is time based, and for 24-hours if you just started populating the graphs there will indeed be emptyness on the left.
good to hear that the script worked though :D
 
caius

caius

Member
@xforce30164 Try running this command to see what types of data the rrds you are producing contain:

Code: 
rrdinfo /var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-local_accepted.rrd

You should see output that shows the various rra types that are contained in the file. There should be lines containing AVERAGE, MIN, MAX. If those are missing then that is why you are having problems.
 
mgunther

mgunther

New Member
xforce30164 said:
Not sure, you can shift graphs, but it is time based, and for 24-hours if you just started populating the graphs there will indeed be emptyness on the left.
Click to expand...
Sorry I wasn't clear. I'm not concerned about the actual graphs and I realise they will fill up as the day goes on. I was talking about the whole "box" with the CPU graph inside it. The CPU box is somehow smaller than the Message Rate box above it. And come to think about it, the Max Range box is also smaller. I would love if they were all the same size and line up perfectly:)
 
caius

caius

Member
In the graph generating script, there are parameters for adjusting the size of the graphs produced. Obj's original script had a couple that were different sizes. I've modified mine to be a bit larger:

Code: 
--width 823 \
--height 217 \

The images produced will be slightly larger than those dimensions due to the labels and key.
 
mgunther

mgunther

New Member
OK, I see what is happening here:
In /home/pi/dump-tools/collectd/make-graphs.sh, all graphs are defined as:
--width 480 \
--height 200 \
But because the first graph (message rate) has both left and right scales it ends up wider than all the others (by an extra 51 pixels). I tried to edit make-graphs.sh and change the width for the rate graph but so far it is staying the same. Not sure why...
 
bramj

bramj

Member
mgunther said:
OK, I see what is happening here:
In /home/pi/dump-tools/collectd/make-graphs.sh, all graphs are defined as:
--width 480 \
--height 200 \
But because the first graph (message rate) has both left and right scales it ends up wider than all the others (by an extra 51 pixels). I tried to edit make-graphs.sh and change the width for the rate graph but so far it is staying the same. Not sure why...
Click to expand...



in directory /var/www/collectd change the index.html like this

Code: 
  <div style="width: 1280px; height: 273px;">
                <img id="rate" width="577" height="273" src="dump1090-rpi-rate-24h.png">
                <img id="acs" width="577" height="273" src="dump1090-rpi-acs-24h.png">
        </div>
        <div style="width: 1280px; height: 273px;">
                <img id="cpu" style="width: 577px; height: 273px;" src="dump1090-rpi-cpu-24h.png">
                <img id="tracks" width="577" height="273" src="dump1090-rpi-tracks-24h.png">
        </div>
        <div style="width: 1280px; height: 273px;">
                <img id="range" width="577" height="273" src="dump1090-rpi-range-24h.png">
                <img id="ctemp" width="577" height="273" src="table-rpi-core_temp-24h.png">
        </div>
        </body>


and in make-graph.sh change

Code: 
local_rate_graph() {
  rrdtool graph \
  "$1" \
  --start end-$4 \
  --width 429 \
  --height 200 \
  --step "$5" \
  --title "$3 message rate" \
  --vertical-label "messages/second" \
  --lower-limit 0  \
  --units-exponent 0 \
  --right-axis 360:0 \
  "DEF:messages=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE" \
  "DEF:strong=$2/dump1090_messages-strong_signals.rrd:value:AVERAGE" \
  "DEF:positions=$2/dump1090_messages-positions.rrd:value:AVERAGE" \
  "CDEF:y2strong=strong,10,*" \
  "CDEF:y2positions=positions,10,*" \
  "LINE1:messages#0000FF:messages received" \
  "AREA:y2strong#FF0000:messages >-3dBFS / hr (RHS)" \
  "LINE1:y2positions#00c0FF:positions / hr (RHS)"
}

and to create new graphs

~/dump-tools/collectd $ sudo ./make-graphs.sh 24h


Screen Shot 2015-04-20 at 21.24.53.png
 
Last edited:
xforce30164

xforce30164

Active Member
bramj said:
in directory /var/www/collectd change the index.html like this

Code: 
  <div style="width: 1280px; height: 273px;">
                <img id="rate" width="577" height="273" src="dump1090-rpi-rate-24h.png">
                <img id="acs" width="577" height="273" src="dump1090-rpi-acs-24h.png">
        </div>
        <div style="width: 1280px; height: 273px;">
                <img id="cpu" style="width: 577px; height: 273px;" src="dump1090-rpi-cpu-24h.png">
                <img id="tracks" width="577" height="273" src="dump1090-rpi-tracks-24h.png">
        </div>
        <div style="width: 1280px; height: 273px;">
                <img id="range" width="577" height="273" src="dump1090-rpi-range-24h.png">
                <img id="ctemp" width="577" height="273" src="table-rpi-core_temp-24h.png">
        </div>
        </body>


and in make-graph.sh change

Code: 
local_rate_graph() {
  rrdtool graph \
  "$1" \
  --start end-$4 \
  --width 429 \
  --height 200 \
  --step "$5" \
  --title "$3 message rate" \
  --vertical-label "messages/second" \
  --lower-limit 0  \
  --units-exponent 0 \
  --right-axis 360:0 \
  "DEF:messages=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE" \
  "DEF:strong=$2/dump1090_messages-strong_signals.rrd:value:AVERAGE" \
  "DEF:positions=$2/dump1090_messages-positions.rrd:value:AVERAGE" \
  "CDEF:y2strong=strong,10,*" \
  "CDEF:y2positions=positions,10,*" \
  "LINE1:messages#0000FF:messages received" \
  "AREA:y2strong#FF0000:messages >-3dBFS / hr (RHS)" \
  "LINE1:y2positions#00c0FF:positions / hr (RHS)"
}

and to create new graphs

~/dump-tools/collectd $ sudo ./make-graphs.sh 24h


View attachment 1703
Click to expand...

That was exactly what I was going to suggest! Thanks @bramj for the quick response =]. (though your width and height in the .sh script don't seem to exactly match your html image settings/values but that would result in some slightly streched graphs (possibly)
and another thing to put on the dynamic script configurator, image size setting
 
You must log in or register to reply here.
Top