Web Portal and Collectd/rrd graphs automated installation

jepolch · May 16, 2015

caius said:
Yes if you are using the same collectd.conf then just copy the RRD directory across. I did that when I reinstalled my pi2 and it worked fine.

Thanks. Good to know. By the way, I was getting the same error Bram mentions above.

caius · May 16, 2015

I suspect that it's because raspbian uses an older version of rrdtool. My pi running raspbian has 1.4.7 which is from 2012 (!) and the pi 2 running arch has 1.5.3.

It's one of the reasons I switched to arch - I got fed up with the ancient software versions that Debian and it's derivatives use.

juha · May 16, 2015

xforce30164 said:

also when I run the script, for the trailing average part it gives me these errors:

Code:

./make-graphs.sh: 59: ./make-graphs.sh: DEF:a=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-local_accepted.rrd:value:AVERAGE:end=now-86400:start=end-86400: not found
./make-graphs.sh: 69: ./make-graphs.sh: DEF:amin=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-local_accepted.rrd:value:MIN:end=now-86400:start=end-86400: not found
./make-graphs.sh: 79: ./make-graphs.sh: DEF:amax=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-local_accepted.rrd:value:MAX:end=now-86400:start=end-86400: not found
./make-graphs.sh: 87: ./make-graphs.sh: DEF:strong=/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-strong_signals.rrd:value:AVERAGE: not found
./make-graphs.sh: 94: ./make-graphs.sh: SHIFT:a:86400: not found
./make-graphs.sh: 118: ./make-graphs.sh: CDEF:7dayaverage=a,b,c,d,e,f,g,+,+,+,+,+,+,7,/: not found
./make-graphs.sh: 122: ./make-graphs.sh: CDEF:min1=amin,bmin,MINNAN: not found
./make-graphs.sh: 137: ./make-graphs.sh: CDEF:maxarea=max,min,-: not found
./make-graphs.sh: 141: ./make-graphs.sh: LINE1:min#FFFF99:mins: not found

even though the file paths seem to be ok.

Did you find a reason for the errors? Having the same problem.
@caius ?

xforce30164 · May 16, 2015

juha said:
Did you find a reason for the errors? Having the same problem.
@caius ?

Hmm, I thought I had posted the solution, but apparently I didn't

.

What I had to do was to remove all the comments between the different parts. (IIRC there was a white-line, comment line and agian white line before each of the line numbers mentioned in your errors.)
Make the code such that it is one constant piece of pure code without the comments. apparently the older RRD version can't handle comments that well.

juha · May 16, 2015

xforce30164 said:
Hmm, I thought I had posted the solution, but apparently I didn't .

What I had to do was to remove all the comments between the different parts. (IIRC there was a white-line, comment line and agian white line before each of the line numbers mentioned in your errors.)
Make the code such that it is one constant piece of pure code without the comments. apparently the older RRD version can't handle comments that well.

Thanks! Worked perfectly.

radiostationx · May 16, 2015

I tried d1090mut and graph install on my pi2, and managed to install all the feeders i wanted.
Things went very well at the start, But
Then I had similar problems to fellow PF member ab cd.
After 3 days, I spent more time diagnosing crashes in mutability including re-installing the whole sd card twice than I did logging and tracking aircraft.

Such a shame as it is a great setup when it was working.
I have reverted back to stock Malcom Robb version of d1090 and all feeders and the station has been fault free "chugging away" for 48hrs with no issues.

As said , such a shame as obj and the guys on this forum have done great things for the adsb monitoring community providing all this marvelous decoder & data collection stuff.
Also it was quite straight forward to set up both mutab & collectd/rrd using the guides and posts on the PF forum (well done all who have contributed) .
Great information exchange on comparing experimentation results Guys !

I give up on mutab/collectd/rrrd ,on my main station, for now anyway, but I do have an experimental spare unit now (pi b) so I will be toying around with that and the software to see if i can get it to work reliablely first.

jepolch · May 16, 2015

xforce30164 said:
Hmm, I thought I had posted the solution, but apparently I didn't .

What I had to do was to remove all the comments between the different parts. (IIRC there was a white-line, comment line and agian white line before each of the line numbers mentioned in your errors.)
Make the code such that it is one constant piece of pure code without the comments. apparently the older RRD version can't handle comments that well.

Yup, you did mention that on your blog.

jepolch · May 16, 2015

For anyone who needs it, I uploaded a copy of my working make-graphs.sh to Dropbox. https://www.dropbox.com/s/u7bt1qvuqp6dn3x/make-graphs.sh

garylovesbeer · May 18, 2015

xforce30164 said:
Jup, I've been noticing that too, not sure what to do about that. I'm planning on writing my own version of the tool, which instead of using python. (which ran very heavy on my raspy, it actually forced some cpu away from dump1090-muta. "top" command showed 55% python, 40% dump-muta and some other stuff, while normally dump-muta is around 50-55%). This is another point for my to-do list though and this last quartile is busier at university again so I have some less time to program than before :/

If you do decide to invest time in writing your own version of this tool is there any way you could make the vertical axis log or something like this to expand the lower part of the graphs as this is where the information I am most interested is ie where obstructions on the horizon are.

Also any way to preserve the previous data and/or time stamp it so an analysis based on time could be done would be useful. Well maybe.

Just a suggestion.

garylovesbeer · May 18, 2015

jepolch said:
For anyone who needs it, I uploaded a copy of my working make-graphs.sh to Dropbox. https://www.dropbox.com/s/u7bt1qvuqp6dn3x/make-graphs.sh

I installed it but I am not getting the graphs updating correctly ie no historical data and no legends across the bottom.

Time for a reinstall I think. Bugger.

Getting good uptimes too.

jepolch · May 18, 2015

garylovesbeer said:
I installed it but I am not getting the graphs updating correctly ie no historical data and no legends across the bottom.

Time for a reinstall I think. Bugger.

Getting good uptimes too.

View attachment 2059

Gary, did you use @xforce30164's web install script? Here's my to-do list for installing the graphing stuff:

Code:

sudo apt-get install rrdtool && sudo apt-get install collectd
wget http://raspi.ayadex.com/volatile/web_install.sh
sudo chmod +x web_install.sh
sudo ./web_install.sh "dump-tools"
cd ~/pi/dump-tools
sudo ./install.sh "crontab"

Also, here's a copy of my /etc/collectd/collectd.conf if you need it. https://www.dropbox.com/s/yozibpk8l2k6a87/collectd.conf?dl=0

garylovesbeer · May 19, 2015

jepolch said:
Gary, did you use @xforce30164's web install script? Here's my to-do list for installing the graphing stuff:

Code:

sudo apt-get install rrdtool && sudo apt-get install collectd wget http://raspi.ayadex.com/volatile/web_install.sh sudo chmod +x web_install.sh sudo ./web_install.sh "dump-tools" cd ~/pi/dump-tools sudo ./install.sh "crontab"

Also, here's a copy of my /etc/collectd/collectd.conf if you need it. https://www.dropbox.com/s/yozibpk8l2k6a87/collectd.conf?dl=0

Thanks for the link.

No this is my original piaware install so it is getting quite ancient. I don't want to muck with it too much as I have customised other aspects eg 2 x DS18B20 temperature probes for case and ambient temperature. Should just grow some and get on with reinstalling raspbian. Bit sick of computers at the moment because my wife's iMac just sh*t itself.

The joy of no real backups...

TomW · May 19, 2015

garylovesbeer said:
my wife's iMac just sh*t itself.

Don't you just hate it when that happens? Also now #1 priority, right?

xforce30164 · May 19, 2015

garylovesbeer said:
Thanks for the link.

No this is my original piaware install so it is getting quite ancient. I don't want to muck with it too much as I have customised other aspects eg 2 x DS18B20 temperature probes for case and ambient temperature. Should just grow some and get on with reinstalling raspbian. Bit sick of computers at the moment because my wife's iMac just sh*t itself.

The joy of no real backups...

Goodluck, you can do it

Also, Im almost readyto release my new version instsller script. (have been rewriting it and even madeit so thatit dynamically grabs code parts based on what each script needs, somewhat comparable to including files.

garylovesbeer · May 23, 2015

xforce30164 said:
Goodluck, you can do it

Just grew some...hope it works...

Screen Shot 2015-05-23 at 4.36.38 pm.png

ab cd · May 28, 2015

If someone has Piaware Image installed, and wants to install and run collectd graphs,
(1) Does he need to install dump1090-mutability, or the already installed version of dump1090 is enough to install & run collectd rrd tools graphs?
(2) If dump1090-mutability installation on Piaware Image is necessary to run collectd, has anything to be done to stop/remove already installed dump1090?

ab cd · Jun 6, 2015

TomW needs help. He has said:
"I got the FA feeder running and started through the steps of xforce30164's auto install script:

Code:

sudo apt-get install rrdtool sudo apt-get install collectd sudo apt-get update sudo apt-get install collectd wget http://raspi.ayadex.com/volatile/web_install.sh sudo chmod +x web_install.sh sudo ./web_install.sh "dump-tools"

The last step creates the following output, with http://192.168.100.7/collectd yielding "ERR_CONNECTION_REFUSED"

Here is the output for the last step.

Code:

[email protected] ~ $ sudo ./web_install.sh "dump-tools" ===== Starting dump-tools installation! ===== ===== Created directories, starting downloads ===== --2015-06-04 11:29:42-- http://raspi.ayadex.com/volatile/dump-tools/install.sh Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086:: Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected . HTTP request sent, awaiting response... 200 OK Length: 3736 (3.6K) [application/x-sh] Saving to: `install.sh' 100%[======================================>] 3,736 --.-K/s in 0s 2015-06-04 11:29:43 (70.1 MB/s) - `install.sh' saved [3736/3736] --2015-06-04 11:29:43-- http://raspi.ayadex.com/volatile/dump-tools/update.sh Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086:: Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected . HTTP request sent, awaiting response... 200 OK Length: 4421 (4.3K) [application/x-sh] Saving to: `update.sh' 100%[======================================>] 4,421 --.-K/s in 0.1s 2015-06-04 11:29:43 (29.6 KB/s) - `update.sh' saved [4421/4421] --2015-06-04 11:29:44-- http://raspi.ayadex.com/volatile/dump-tools/collectd/co llectd.conf Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086:: Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected . HTTP request sent, awaiting response... 200 OK Length: 2586 (2.5K) [text/plain] Saving to: `collectd.conf' 100%[======================================>] 2,586 --.-K/s in 0s 2015-06-04 11:29:44 (72.1 MB/s) - `collectd.conf' saved [2586/2586] --2015-06-04 11:29:44-- http://raspi.ayadex.com/volatile/dump-tools/collectd/du mp1090.db Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086:: Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected . HTTP request sent, awaiting response... 200 OK Length: 235 [text/plain] Saving to: `dump1090.db' 100%[======================================>] 235 --.-K/s in 0s 2015-06-04 11:29:44 (5.98 MB/s) - `dump1090.db' saved [235/235] --2015-06-04 11:29:44-- http://raspi.ayadex.com/volatile/dump-tools/collectd/du mp1090.py Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086:: Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected . HTTP request sent, awaiting response... 200 OK Length: 7543 (7.4K) [text/plain] Saving to: `dump1090.py' 100%[======================================>] 7,543 --.-K/s in 0.1s 2015-06-04 11:29:45 (51.0 KB/s) - `dump1090.py' saved [7543/7543] --2015-06-04 11:29:45-- http://raspi.ayadex.com/volatile/dump-tools/collectd/ma ke-graphs.sh Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086:: Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected . HTTP request sent, awaiting response... 200 OK Length: 7136 (7.0K) [application/x-sh] Saving to: `make-graphs.sh' 100%[======================================>] 7,136 --.-K/s in 0.1s 2015-06-04 11:29:45 (46.6 KB/s) - `make-graphs.sh' saved [7136/7136] ===== Downloads complete, starting installation ===== ===== Starting base installation of dump-tools! ===== Reading package lists... Done Building dependency tree Reading state information... Done collectd is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded. --2015-06-04 11:29:55-- http://raspi.ayadex.com/volatile/dump-tools/collectd/co llectd.conf Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086:: Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected . HTTP request sent, awaiting response... 200 OK Length: 2586 (2.5K) [text/plain] Saving to: `collectd.conf' 100%[======================================>] 2,586 --.-K/s in 0s 2015-06-04 11:29:56 (59.8 MB/s) - `collectd.conf' saved [2586/2586] [ ok ] Restarting statistics collection and monitoring daemon: collectd. ls: cannot access /var/lib/collectd/rrd/localhost/dump1090-rpi: No such file or directory ===== If files are displayed above, the collectd/rrd installation completed succ esfully! ===== ===== Starting webUI installation! ===== mkdir: cannot create directory `/var/www/collectd': No such file or directory ./install.sh: 50: cd: can't cd to /var/www/collectd --2015-06-04 11:32:00-- http://raspi.ayadex.com/stable/libs/jquery.js Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086:: Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected . HTTP request sent, awaiting response... 200 OK Length: 95931 (94K) [application/javascript] Saving to: `jquery.js' 100%[======================================>] 95,931 --.-K/s in 0.002s 2015-06-04 11:32:13 (38.6 MB/s) - `jquery.js' saved [95931/95931] --2015-06-04 11:32:13-- http://raspi.ayadex.com/volatile/dump-tools/collectd/in dex.html Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086:: Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected . HTTP request sent, awaiting response... 200 OK Length: 1623 (1.6K) [text/html] Saving to: `index.html' 100%[======================================>] 1,623 --.-K/s in 0s 2015-06-04 11:32:14 (23.6 MB/s) - `index.html' saved [1623/1623] ===== Finished webUI installation, creating initial graphs! ===== 6576 (process ID) old priority 0, new priority 5 ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_aircraft-r ecent.rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_cpu-demod. rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_tracks-all .rrd': No such file or directory 577x259 ERROR: Could not save png to '/var/www/collectd/table-rpi-core_temp-24h.png' ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_range-max_ range.rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-l ocal_accepted.rrd': No such file or directory 6585 (process ID) old priority 0, new priority 5 ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_aircraft-r ecent.rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_cpu-demod. rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_tracks-all .rrd': No such file or directory 577x259 ERROR: Could not save png to '/var/www/collectd/table-rpi-core_temp-7d.png' ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_range-max_ range.rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-l ocal_accepted.rrd': No such file or directory 6594 (process ID) old priority 0, new priority 5 ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_aircraft-r ecent.rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_cpu-demod. rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_tracks-all .rrd': No such file or directory 577x259 ERROR: Could not save png to '/var/www/collectd/table-rpi-core_temp-30d.png' ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_range-max_ range.rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-l ocal_accepted.rrd': No such file or directory 6603 (process ID) old priority 0, new priority 5 ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_aircraft-r ecent.rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_cpu-demod. rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_tracks-all .rrd': No such file or directory 577x259 ERROR: Could not save png to '/var/www/collectd/table-rpi-core_temp-365d.png' ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_range-max_ range.rrd': No such file or directory ERROR: opening '/var/lib/collectd/rrd/localhost/dump1090-rpi/dump1090_messages-l ocal_accepted.rrd': No such file or directory ===== Finished creating inital graphs! ===== First part of installation is finished. please check http://<rpi-ip>/collectd an d see if you see any graphs. If that is the case please run sudo ./install.sh "crontab" from the /home/pi/dum p-tools diretory to enable automatic graph creation. [email protected] ~ $

Thanks for your assistance!

Tom"

bramj · Jun 7, 2015

There is a problem with restarting the collected deamon.
maybe the problem is the hostname off the raspberry , its change to "pibeta"

---------------------------------
2015-06-04 11:29:56 (59.8 MB/s) - `collectd.conf' saved [2586/2586]

[ ok ] Restarting statistics collection and monitoring daemon: collectd.
ls: cannot access /var/lib/collectd/rrd/localhost/dump1090-rpi: No such file or directory
===== If files are displayed above, the collectd/rrd installation completed succ esfully! =====
===== Starting webUI installation! =====
mkdir: cannot create directory `/var/www/collectd': No such file or directory
./install.sh: 50: cd: can't cd to /var/www/collectd
--2015-06-04 11:32:00-- http://raspi.ayadex.com/stable/libs/jquery.js
Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086::
Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected .
HTTP request sent, awaiting response... 200 OK
Length: 95931 (94K) [application/javascript]
Saving to: `jquery.js'

TomW · Jun 7, 2015

bramj said:
There is a problem with restarting the collected deamon.
maybe the problem is the hostname off the raspberry , its change to "pibeta"

---------------------------------
2015-06-04 11:29:56 (59.8 MB/s) - `collectd.conf' saved [2586/2586]

[ ok ] Restarting statistics collection and monitoring daemon: collectd.
ls: cannot access /var/lib/collectd/rrd/localhost/dump1090-rpi: No such file or directory
===== If files are displayed above, the collectd/rrd installation completed succ esfully! =====
===== Starting webUI installation! =====
mkdir: cannot create directory `/var/www/collectd': No such file or directory
./install.sh: 50: cd: can't cd to /var/www/collectd
--2015-06-04 11:32:00-- http://raspi.ayadex.com/stable/libs/jquery.js
Resolving raspi.ayadex.com (raspi.ayadex.com)... 81.169.145.86, 2a01:238:20a:202 :1086::
Connecting to raspi.ayadex.com (raspi.ayadex.com)|81.169.145.86|:80... connected .
HTTP request sent, awaiting response... 200 OK
Length: 95931 (94K) [application/javascript]
Saving to: `jquery.js'

What should the hostname be? Does that mean it is not possible to have this software running on multiple devices? Sorry for the noobish questions...
Tom

bramj · Jun 7, 2015

TomW said:
What should the hostname be? Does that mean it is not possible to have this software running on multiple devices? Sorry for the noobish questions...
Tom

There are some issue's reported in this thread.
Web Portal and Collectd/rrd graphs automated installation

"The hostname it's hardcoded into the installer. The path in my case should be /var/lib/collectd/rrd/banana/dump1090-rpi/. Because of this, the installer didn't even copy/generate the rrd's. /var/lib/collectd/rrd/banana/ does not contain the dump1090-rpi folder.

/var/lib/collectd/rrd/$HOSTNAME/ would make the job."

Web Portal and Collectd/rrd graphs automated installation

Active Member

Member

New Member

Active Member

New Member

Member

Active Member

Active Member

New Member

New Member

Active Member

New Member

Member

Active Member

New Member

Senior Member

Senior Member

Member

Member

Member