[Dprglist] Visualizing data [was: PID]

[Murray Altheim]

On 4/10/20 5:33 am, David Anderson via DPRGlist wrote:
[...]
> It's actually pretty trivial to use gnuplot if you are comfortable
> with Linux.  Here's what I did to plot your log data. First, data
> from your html file were cut and pasted to a flat ASCII file named
> "murray_log-20200929.txt."

Hi David,

I've been a daily Linux user since the mid-90s, and UNIX before that,
so the command line is a happy place for me.

> Then, because Linux uses white space as a column separator by default, a few places had to be modified which was done like this:
> 
> $ cat murray_log-20200929.txt | grep error | sed 's/=/= /g'
> 
> which passes through the lines which have the word "error" in them and extends the "error=" and "D=" labels by an extra space so they are seen by the parser as separate fields.

Ahh yes.

> That output was piped to a gawk command that printed out only the data values of interest and not the accompanying ASCII labels:
> 
> $ cat murray_log-20200929.txt | grep error | sed 's/=/= /g' | gawk '{print $1,$2,$7,$9,$11,$13,$15,$17}'
> 
> Then a separate file was created for the left and right motors by grep'ing on the terms "pid:port" and "pid:stbd" so the whole thing for the port motor looks like this:
> 
> $ cat murray_log-20200929.txt | grep error | sed 's/=/= /g' | gawk '{print $1,$2,$7,$9,$11,$13,$15,$17}' | grep "pid:port" > port.dat
> 
> The process of graphing these data with gnuplot is then just a matter of plotting the fields of interest with gnuplot's "plot" command:
> 
> gnuplot> plot "port.dat" using ($1):($3) title "Error" with lines, \
>          "port.dat" using ($1):($4) title "P " with lines, \
>          "port.dat" using ($1):($7) title "Setpt" with lines, \
>          "port.dat" using ($1):($8) title "Output" with lines
> 
> Each of these lines uses the first field, the line number, as the horizontal X value, and the given field as the Y value.  If the log file is created with a time stamp as the first entry and white space (or commas) as the field separators, 
> then none of the data preprocessing steps would be necessary,  and the graphs could be generated by gnuplot directly from the log files with just a single command, as above.

Well, I may be a daily Linux user and familiar with all those tools, but
you've still made quick work of it, very nice job. Seeing it all laid out
like that is very helpful.

I'm trying to think whether to go with the Grafana idea or GnuPlot at this
point, as Grafana is very good at plotting time series data.

   https://grafana.com/docs/grafana/latest/getting-started/timeseries/

I had a little experience with this kind of thing when working at MetService,
the NZ weather service. Shame I didn't jump into more projects that would
have amplified those skills. There were some serious experts there who could've
lent a hand.

I've cleaned up my log now as per Chris' recommendation, so I'm generating
clean CSV with millisecond timestamps and all the data for both PIDs on one
line. I started on this journey just printing it in color to sysout so I
could "manually" view what was happening, but obviously that's nowhere near
as useful as actually graphing it, especially when we're trying to compare
multiple values, and looking at this motion profile idea on my "one meter
behaviour".

It's a Sunday here and for a change not raining (though we're expecting
gales but my back yard is sheltered a bit) so I may have to do some firewood
chopping and get out the power tools and start to put together my cypress
wood garden boxes. Otherwise Spring will be over before we ever plant any
vegetables... so hopefully later I can do a bit more robotic works.

Oh, an almost pristine copy of the 1st edition Jones/Flynn "Mobile Robots:
Inspiration to Implementation" book arrived in the mail yesterday, thanks
for the recommendation. It turns out I had seen this book before, I think
I'd checked out a copy from the Wellington Central Library (now condemned
and to be bowled due to severe damage from the last earthquake). As it's
from 1993 the tech is old school but the ideas still very useful. I don't
think when I'd looked at it before the concept of a PID controller would
have had any currency, funny it's central to getting my robot running now.

Maybe I've learned a bit.

Cheers,

Murray

...........................................................................
Murray Altheim <murray18 at altheim dot com>                       = =  ===
http://www.altheim.com/murray/                                     ===  ===
                                                                    = =  ===
     In the evening
     The rice leaves in the garden
     Rustle in the autumn wind
     That blows through my reed hut.
            -- Minamoto no Tsunenobu