Stoker monitoring with open software and/or Linux


 
Bryan, I hope you are still watching this thread. I have adapted a perl program that fetches the json data from stoker and formats it in csv with the 1st two values for each record (flatfile, is that 1st normal form?) being the date and time of the poll for that record. I have pretty much decided what other data I want in the csv file, so I have been looking back at the gnuplot stuff. It would help me to understand your gnuplot setup if I had a format description for each line of your datafile, with a couple of records as a sample. Particularly I am having trouble figuring out how to use a filled line for the moving average of my blower status. I tried to add 'with filledcurves' to my Bott.Vent% plot in my gnuplot setup and gnuplot is filling the curve above rather than from the x axis to the line. It is also plotting against y rather than against y2. It is including the filled line in the key the way I did it.
 
Yeah sure! I know it gets a little confusing
<pre class="ip-ubbcode-code-pre">
# sample data
2010-09-05 02:36:10-04:00,225,225.4,55.2,48.4,95.6,3,8,0
2010-09-05 02:36:23-04:00,225,225.4,55.2,48.4,95.8,0,7,0
2010-09-05 02:36:42-04:00,225,225.1,55.2,48.4,95.6,8,6,0
2010-09-05 02:37:01-04:00,225,225.1,55.2,48.4,95.6,3,6,0
2010-09-05 02:37:11-04:00,225,225.1,55.2,49.2,95.8,0,6,0
2010-09-05 02:37:21-04:00,225,225.1,56.0,49.2,95.6,0,5,0
2010-09-05 02:37:34-04:00,225,224.8,56.0,49.2,95.6,9,5,0
# Fields are
DateTime,SetPoint,Probe1,Probe2,Probe3,Ambient,FanInstantaneous,FanMovingAverage,LidDetectIndicator

plot "data.txt" \

# I wanted to fill the area between the SetPoint and the Pit value, so you tell gnuplot
# to plot Time:point1:point2 with filledcurves and it fills the area between Point1 and
# Point2, in this case with a gray
using 1:3:2 linecolor rgb "#eeeeee" notitle with filledcurves, \

# Next is the yellow lid indicator. Note that solid fills are not transparent
# so put the least important filled stuff first, and work toward the more
# important filled boxes. If you can get transparency working in the PNG
# terminal, I'd love to hear how
'' using 1:9 linecolor rgb "#ffffcc" notitle axis x1y2 with boxes fs solid, \

# this is the Moving Average Fan speed in a filled blue bar. You could do also
# do a "with filledcurves above y1=0" instead of "with boxes fillstyle solid"
# (I think that's the correct filledcurves syntax)
'' using 1:8 linecolor rgb "#cceeff" notitle axis x1y2 with boxes fs solid, \

# The instantaneous fan speed, I think I deleted this line before I generated
# my example graph, because it doesn't have this dark blue line on it so ignore!
'' using 1:7 linecolor rgb "#3169c6" notitle axis x1y2, \

# The Set Point in a thin red line
'' using 1:2 linecolor rgb "red" notitle, \

# The pit temperature in a thicker red line
'' using 1:3 linecolor rgb "red" linewidth 3 title "Pit", \

# Finally the food probes and ambient temperature
'' using 1:4 linecolor rgb "green" title "Brisket", \
'' using 1:5 linecolor rgb "blue" title "Pit Top", \
'' using 1:6 linecolor rgb "purple" title "Ambient"
</pre>

EDIT: Note I don't think you can have comments and spaces in your gnuplot config, so don't try to use this commented version directly.
 
Originally posted by Bryan Mayland:
Yeah sure! I know it gets a little confusing
<pre class="ip-ubbcode-code-pre">
# sample data
2010-09-05 02:36:10-04:00,225,225.4,55.2,48.4,95.6,3,8,0
2010-09-05 02:36:23-04:00,225,225.4,55.2,48.4,95.8,0,7,0
</pre>
...
# Fields are
DateTime,SetPoint,Probe1,Probe2,Probe3,Ambient,FanInstantaneous,FanMovingAverage,LidDetectIndicator
Click to expand...
Thanks. That makes it much easier to figure out your plot directives.

# I wanted to fill the area between the SetPoint and the Pit value, so you tell gnuplot
# to plot Time:point1:point2 with filledcurves and it fills the area between Point1 and
# Point2, in this case with a gray
using 1:3:2 linecolor rgb "#eeeeee" notitle with filledcurves, \

# Next is the yellow lid indicator. Note that solid fills are not transparent
# so put the least important filled stuff first, and work toward the more
# important filled boxes. If you can get transparency working in the PNG
# terminal, I'd love to hear how
'' using 1:9 linecolor rgb "#ffffcc" notitle axis x1y2 with boxes fs solid, \

# this is the Moving Average Fan speed in a filled blue bar. You could do also
# do a "with filledcurves above y1=0" instead of "with boxes fillstyle solid"
# (I think that's the correct filledcurves syntax)
'' using 1:8 linecolor rgb "#cceeff" notitle axis x1y2 with boxes fs solid, \
Click to expand...

That should help a lot.
...
Do you happen to remember anything in gnuplot that calculates and plots a moving average? Otherwise I am going to have to do it in perl, since stoker.json returns either 0 or 1.

On a related question, do you really think you need to poll in such a tight loop? (sleep 60). I was thinking sleep 300 maybe.

EDIT: Note I don't think you can have comments and spaces in your gnuplot config, so don't try to use this commented version directly.
Click to expand...

That is one reason why I intend ultimately to fill the config file in perl and feed it to gnuplot stdin after it is all created. That way I will also be able to take input from the commandline. Also, if the first line of the data sent to gnuplot consists of a list of labels, those can be used to label the graph. I am collecting the names for the different probes and fan that the user has set with the stoker interface, since those are part of the json structure.
 
Curt, or anybody else:

I am now able to generate a series of flat, (1st normal form?) records of the form:
<pre class="ip-ubbcode-code-pre">
$VAR1 = [
'
',
'2010-09-08,17:27:42,',
'Blower,0,',
'Fire,65,69.3',
'Food1,170,69',
'Food2,32,69',
'Food3,32,67.9'
];
</pre>

It looks like the record starts with an empty newline. Is that what I am seeing above? Is that a problem? I can't see where the empty record is being generated, unless the foreach loop is running empty the first time through. I also still do not see how the fieldsort in the slightly modified version of the code that I posted previously is sorting the "line" the way it is.

I am not really a programmer, just a very diligent empiricist. I couldn't follow a flow chart if my life depended on it.

Is part of the problem that I am not really completely normalized?
 
Stew,

Possibly the first '@sensors' value in the foreach loop has null values. Use Dumper on each @sensors result to determine what is being returned.

Debugging through a forum is a real challenge.

Curt
 
I couldn't find anything in gnuplot that did the moving average. I did look around at their curve fitting functions which didn't help. I don't know if it is possible but I came up empty.

I actually poll tighter than that, sleep 30. My BBQ Controller is a homemade project, so I'm still interested in tuning the parameters of the controller loop. However, on my system the period of the temperature oscillations (the time it takes for the temp to go from 223 to 227 and back to 223) is about 5 minutes, so you'd want at least 5 samples per period to be able to see the shape of the wave. If I only sampled every 5 minutes it could look like the temperature is always 223.

Not that I *need* to know that the temperature I see is +/-4 degrees... but yes, I need to know!
icon_smile.gif
 
Curt,

I did as you suggested and it did not show the \n in the sensors loop, but only in the dump of sorted. I was pushing a \n onto the completed unsorted array_of_sensors, I think. Then the fieldsort routine was sorting that empty value to the beginning of the array. I know I came up with the sort module on my own so you may not know anything about how he does it, but does my analysis above make sense?

Regarding debugging through a bb, I agree. However I have been reluctant to try to email you since you do not expose your email here, on bbqmonitor or on bbqbuds. I also do not like to expose my email, and I avoid social networking sites like the plague. In truth I liked the Internet better in the days when I first encountered Unix and Usenet News at the age of 38 in 1985. In those days there would be a few hundred of us reading a usenet group or a listserve. Of course, if you make a living programming for the modern web universe, then your point of view would probably be different.
icon_smile.gif


Bryan,

I guess I will probably poll sleep 60 when I do my 1st cook with the stoker this afternoon. I may have to poll tighter than that to get a meaningful pattern of 0's and 1's from the blower. For now I will just collect the data with my get-stoker-json.pl program from an sh script while loop; sleep 60 in the background. then I will monitor it with tail -f every once in a while. I will probably cheat and have stokerlog running on my Win XP video-only computer.

Either of you,

Now both the dump of the array and the ASCII have a comma between each element, but also a comma at the end of what I think of as a line. I do not think it will cause any problem with how I plan to use it as a csv file, but if I were going to try to use the array in any other way would the terminating comma cause problems?

I will have more to report tomorrow.
 
I smoked some baby backs yesterday afternoon. Got started late so I started at 275, but in a few minutes decided to up the pit temp to 300 (did it with the builtin webserver). Logged by calling my perl json to csv program from shell with sleep 60. Ribs passed the flop test at 2 1/2 hrs.. Pulled fan lead on the stoker to shut it off, and the stoker immediately deleted it from the json report. Kept logging to watch the temperature profile of the pit. My conversion program did not pad the blower name field and on field with ,, so my logfile was off by two fields from that point on. It was easy to fix the small number of lines by hand with an editor, but that is obviously not an acceptable solution.

Here is the section of the code that adds the blower info to the line:
<pre class="ip-ubbcode-code-pre"> # iterate over each blower in the JSON structure:
my $blowers = 1;
foreach my $blowers(@{$json_text->{stoker}->{blowers}}){
# create an array of blowers information:
@blower = ("$blowers->{name},$blowers->{on},");
push(@array_of_sensors , @blower);
$blowers++;
}
</pre>

I obviously need to do something if there is no blower but am not sure what. How do I test on the return of the foreach $blowers loop and do an else of pushing two empty fields onto the array_of_sensors?

I still haven't figured out scope in perl, so have to run this without 'strict'. One thing at a time.
 
Here is the plot of yesterday's cook:
100909bbribs2.png


I am still calling csv to plot in a shell script.
logger-perl-script>file.csv; gnuplot-config-and-execute-sh-script .

The y2 axis of the 0's and 1's works well enough that I am not going to bother with a moving average.

I was able to label the plotted lines by putting an initial column label line at the top of the data file and then titling the plot calls with column(N), which was documented in gnuplot 4.2.5 that was installed by ubuntu karmic. I had to go bleeding edge and get perl 4.4.4 from a non-ubuntu repository before it would work. This is an important feature so you don't have to hardcode any more than absolutely necessary in the perl config.

The ultimate goal is to incorporate all the steps into a perl program. One step at a time, since I cannot do flow charts.
icon_rolleyes.gif
 
Originally posted by Stew Ellis:

Here is the section of the code that adds the blower info to the line:
<pre class="ip-ubbcode-code-pre"> # iterate over each blower in the JSON structure:
my $blowers = 1;
foreach my $blowers(@{$json_text->{stoker}->{blowers}}){
# create an array of blowers information:
@blower = ("$blowers->{name},$blowers->{on},");
push(@array_of_sensors , @blower);
$blowers++;
}
</pre>

I obviously need to do something if there is no blower but am not sure what. How do I test on the return of the foreach $blowers loop and do an else of pushing two empty fields onto the array_of_sensors?
Click to expand...

My Perl is rusty, but seems to me you should be able to do a count operation on $json_text->{stoker}->{blowers} before you enter that loop.

joe
 
Originally posted by Stew Ellis:

<pre class="ip-ubbcode-code-pre"> # iterate over each blower in the JSON structure:
my $blowers = 1;
foreach my $blowers(@{$json_text->{stoker}->{blowers}}){
# create an array of blowers information:
@blower = ("$blowers->{name},$blowers->{on},");
push(@array_of_sensors , @blower);
$blowers++;
}
</pre>
Click to expand...

Try this:
<pre class="ip-ubbcode-code-pre">
# iterate over each blower in the JSON structure:
my $blower_count = 0;
foreach my $blowers(@{$json_text->{stoker}->{blowers}}){
# create an array of blowers information:
@blower = ("$blowers->{name},$blowers->{on},");
push(@array_of_sensors , @blower);
$blower_count++;
}

while ($blower_count < 1)
{
push(@array_of_sensors , ',,') ;
$blower_count++ ;
}
</pre>

Curt
 
Thank you Joe and Curt and Bryan for putting up with my lack of chops. I will probably not get around to playing with the Stoker again until Monday if my wife is tired of bbq, or Sunday if she is ready for more. In the last month we fed on a brisket for a week, a butt for another week and a turkey breast for a week and a half.

When I was learning C to the degree I ever did back in the '80's I did most of my work in jove, which was about as complete an EMACS-alike editor as would run on 286 Xenix.
I still use it as my primary editor.

Jove supports ctags and allows one to step through a program, but it has been a long time since I did anything like that. That and a lot of printf's. Does either of you use anything like ctags to step through code? I don't even remember if it did anything except step to external functions.
 
I included the counter almost exactly the way Curt suggested and it fixed THAT problem. Because of the way Sort::Fields works, however, I had to instantiate blower name, other wise each line began with ,, and my plot script requires all probes and blowers to be in alpha order with blower first. Since stoker is not instantiating that value when I unplug blower to see how long it takes the fire to die, I used <pre class="ip-ubbcode-code-pre">push(@array_of_sensors , 'Absent_Blower,,') ;</pre>
That will sort before most any conceivable name for the blower.

It would be easier if stoker always returned completely normalized lines of data for all sensors and blowers in a canonical order.

Thanks to both of you.
 
I know Curt said it is hard to debug over a BB...

Now I have another problem with perl. I am trying to put the json formatting loop into a 'while (1)' loop to continuously poll the stoker until the program is interrupted by the user. So far I had done this with a shell script:
<pre class="ip-ubbcode-code-pre">
#!/bin/sh

while [ 1 ] ; do
perl ./get-stoker-json.pl >> stoker-$$.log
sleep 60
done
</pre>

I should be able to do that with perl, right?

If I want the output to go to STDOUT I do not have any problem except the loop going as fast as the code will run. If I 'open(FH)' and 'print FH "@data \n";' then the data are printed to the file, but again as fast as the loop can run. As soon as I add a sleep call anywhere in either inner or outer loops I can explicitly 'print STDOUT @data' or do a Dump of the array to stdout, but the 'print FH @data' does not take place. Take the sleep out and both FH and STDOUT are written to, sleep back in FH is not written to but STDOUT still gets written to.

I wrote a simple script to see if I was getting lost in my json decoder. The simple script perfectly replicates the problem as described above.

<pre class="ip-ubbcode-code-pre">#!/usr/bin/perl -w

use strict;
use Data::Dumper;
$Data::Dumper::Indent = 1 ;

open(OUT, "> whileprint.txt") || die "Could not open whileprint.txt" ;
select(OUT);
#$outfile = "whileprint.txt";
#my $i = 1;
my @data = ();

#print "before outer while\n";
while (1){
my $i = 1;
@data =();
# print "in outer while, i is $i\n";
while ($i < 10){
# print "in inner while, i is $i\n";
push(@data , "$i,");
# print "@data \n";

$i++;
}
print @data;
print STDOUT Dumper (\@data);
sleep 5;
# print OUT "@data after while loop\n";
}
#print OUT "Cancelled";
</pre>
I have tried with the sleep arg in (), no difference. I have tried printing to OUT and doing the select(OUT) version. Again, no difference. I obviously have write permissions or it would not work without the sleep.

I have tied googling for several hours but have not been able to find anything that helps me on this. I really want to do this in perl rather than sh.

5:19P EST. I did try the commented out filename above. In further experiments, if I open the file to STDOUT, then all my prints go to the file the way I want. That does not make sense to me. So now it works, but I like to understand why the above did not.
 
print @data;
print STDOUT Dumper (\@data);
Click to expand...

I'm not sure of everything you did but the output from both lines of code goes to the same place (STDOUT, the default). If you need a separate output use STDERR for non-data output.

You shouldn't have to use any select's in this application.

Curt
 
I was banging my head against the computer so hard yesterday that I missed your post. I got things cleaned up so I am back to 'use strict' with warnings on. It works by opening $logfilename for writing assigned to STDOUT and anything I want the user to see, such as "no stoker at IP_ADDR" or "output to $logfilename"
I am printing to STDERR.

I wasted several hours yesterday finding out that perldoc is wrong or extremely unclear about how getopts stores arguments that are passed on the commandline. After going through about a dozen online tuts I found one that worked, so I am now getting hvo:i: from commandline switches and giving a help prompt, setting more verbose output to STDERR, letting the user name her own outfile or passing the stoker IP_ADDR. I might add a switch for whether the user wants output to a file or to the terminal. That is why I wanted to do a select(), so that I could redirect stdout conditionally to term or file.

It is now almost 180 lines long and I still do not have all the verbose output conditioned to the state of the -v switch.

What is the best way to post it? It seems a little long to post to this forum.

Now, I need to recode the stoker-plotter.sh module from sh to perl.


It has gr
 
Originally posted by Stew Ellis:
I might add a switch for whether the user wants output to a file or to the terminal. That is why I wanted to do a select(), so that I could redirect stdout conditionally to term or file.
Click to expand...

Here is a simple way to handle that:

use Getopt::Long ;
my $OUT ;
my $OutFileName = '' ;

GetOptions
(
'outfile=s' => \$OutFileName ,
)
or die ("Getoption $!") ;

if ($OutFileName)
{
open ($OUT, ">> $OutFileName") ;
}
else
{
$OUT = *STDOUT ;
}

## $OUT is either the output file or STDOUT

Curt
 
Doing things explicitly, if I have the code
<pre class="ip-ubbcode-code-pre">
my $OUT;
open($OUT, ">$logfilename") or die "Can't open output";
</pre>

followed by the loops that collect one time sample of the stoker, then
<pre class="ip-ubbcode-code-pre">
my @sorted = fieldsort ',', [1, 2, 4], @array_of_sensors;
print $OUT "@sorted\n";
sleep (5);
</pre>

nothing is printed to anything. If I leave the $OUT out of the print statement, then the output goes to STDOUT (terminal) but not to anything else.

If I add a 'select($OUT) after the 'open($OUT)... still no output gets printed to anything. Only if I open the file to STDOUT do I get any output to the file.
 
Stew

Your problem may be output buffering.
Place the following code after the open:
<pre class="ip-ubbcode-code-pre"> $| = 1 ;</pre>
this will flush the buffer after each print command.

Curt
 
You must log in or register to reply here.

 

Back
Top