[Dprglist] Visualizing data [was: PID]
Christian M Netter
netterchris at gmail.com
Sat Oct 3 06:23:45 PDT 2020
Why not just use MS Excel or one of the free alternatives? Or are you trying to graph the data in real time while the robot is running?
Chris
On Oct 3, 2020, at 5:40 AM, Murray Altheim via DPRGlist <dprglist at lists.dprg.org> wrote:
Hi Bob,
As you may imagine, time to do things always seems too short. I had a
go at installing both Grafana and Elasticsearch via Docker containers
onto a "spare" NVidia Xavier NX I'd bought and unfortunately haven't
been able to use for its intended purpose [I've tried 4-5 times to
install the Cuda nvGRAPH library but the whole thing then fails to
boot and I've just given up on this $800 or so of investment, a real
regret, but that's another story...].
I was able to create Grafana and Elasticsearch accounts, etc. but
haven't managed to figure out how to tie them together with the ES as
a data source yet. This involves creating a data source and pointing
Grafana at Elasticsearch, but I haven't got that far.
My Logger class extends the Python logging library, and I just made a
minor change to now write just the PID data to a CSV compatible file.
My new data output is as per David, Karim and others' suggestions
(thank you!) now running on its own thread an entirely separate 20Hz
loop (i.e., separate from the PID controller's own loops) that polls
*all* of the data and writes it onto one line of the log.
I then attempted to use the Python-to-Elasticsearch library and wrote
a test class but for some reason the ping() feature failed to connect
to the server (via IP address), so I gave up and began working once
again on the log output from my PID controllers. At least that's done,
it even prints out a header and legend:
name : description
kp : proportional constant
ki : integral constant
kd : derivative constant
p.cp : port proportial value
p.ci : port integral value
p.cd : port derivative value
p.lpw : port last power
p.cpw : port current motor power
p.spwr : port set power
p.cvel : port current velocity
p.stpt : port velocity setpoint
s.cp : starboard proportial value
s.ci : starboard integral value
s.cd : starboard derivative value
s.lpw : starboard proportional value
s.cpw : starboard integral value
s.spw : starboard derivative value
s.cvel : starboard current velocity
s.stpt : starboard velocity setpoint
p.stps : port encoder steps
s.stps : starboard encoder steps
https://service.robots.org.nz/wiki/attach/PIDController/ros-2020_10_02T20_31_53_278210.csv
David Anderson seems to have a (as an outside observer) very quick process to
get data into GnuPlot, or maybe just a ton of experience. I'd used GnuPlot on
my v1 PID controller but would both as a hobbiest and software professional
love to be able to populate an Elasticsearch DB and automatically get that
into Grafana. I have no need for history so overwriting the ID of the DB
record would be fine.
If you've got some relatively easy way to get CSV into Elasticsearch I'd be
quite happy to learn how.
Cheers,
Murray
> On 3/10/20 1:46 pm, Bob Cook via DPRGlist wrote:
> HI Murray,
> Grafana does pretty slick graphs without much work at all. Here is an example image of a (not very good) PID implementation showing the target speed, actual speed, and error. As you can see this is going from speed=0 to speed=20 then speed=0 again after 30 seconds.
> Happy to share the detail of my Docker setup if that would help. I’m using Docker Compose to create three containers: ElasticSearch, Kibana, and Grafana. The three containers share a private network, and the data for each resides on the host rather than inside the containers themselves - makes upgrading the containers easy without losing the data. Sort of a beautiful thing made simple by Docker.
> - Bob
>>> On Oct 1, 2020, at 1:24 AM, Murray Altheim via DPRGlist <dprglist at lists.dprg.org <mailto:dprglist at lists.dprg.org>> wrote:
>>
>> On 1/10/20 9:25 am, Bob Cook via DPRGlist wrote:
>>> Hi Murray,
>>> You may want to look at Grafana as a tool to visualize time series
>>> data. It is open source and easy to set up if you are familiar with
>>> Docker containers. It pulls data from a variety of sources, I’m using it with an Elasticsearch instance, also as a Docker container.
>>> I set them up on a Linux host with Docker, but apparently you can
>>> use Windows or macOS as your Docker host pretty easily too.
>>
>> Hi Bob,
>>
>> I've as part of various jobs designed and implemented both Solr and
>> ElasticSearch services but I've never tried standing up a Docker
>> instance nor have I used Grafana, though I've looked over the website
>> a few times. I'm a Linux user so yeah, it doesn't sound like it'd be
>> too difficult to attempt what you've done, maybe easier than trying
>> to do it the hard way, which is kinda what I'd done on my version 1
>> PID controller: write the data from the PID to a file on the robot,
>> then either on a HDMI-connected monitor to the Raspberry Pi or on my
>> workstation computer doing the GnuPlot visualisation. That took a
>> fair bit of effort so if Grafana is substantially easier and more
>> configurable I might give it a try. Good learning experience both for
>> my robotics as well as job skills too.
>>
>>> I’m using Python to push data into the ES instance. There is a Python
>>> lib for easy access to ES. My robot records a binary stream of stats
>>> that I convert and upload offline, after I’ve collected data from a test run.
>>
>> Wow, thanks for that clue -- Python to ElasticSearch would be a pretty
>> huge simplification if the ElasticSearch to Grafana connection were
>> similarly easy. But again, a good learning experience in either case.
>>
>> If I were doing a microservice architecture between powerful services
>> on a fast network I might want to send all the log messages directly
>> to ElasticSearch, but considering this is running on a Raspberry Pi
>> over WiFi and I don't need the data to be live-streamed, I'll likely
>> think about how to load a completed log (as CSV, as Chris suggested)
>> after the fact.
>>
>> Thanks much,
>>
>> Murray
>>
>> ...........................................................................
>> Murray Altheim <murray18 at altheim dot com> = = ===
>> http://www.altheim.com/murray/ === ===
>> = = ===
>> In the evening
>> The rice leaves in the garden
>> Rustle in the autumn wind
>> That blows through my reed hut.
>> -- Minamoto no Tsunenobu
>>
>> _______________________________________________
>> DPRGlist mailing list
>> DPRGlist at lists.dprg.org <mailto:DPRGlist at lists.dprg.org>
>> http://lists.dprg.org/listinfo.cgi/dprglist-dprg.org
> _______________________________________________
> DPRGlist mailing list
> DPRGlist at lists.dprg.org
> http://lists.dprg.org/listinfo.cgi/dprglist-dprg.org
--
...........................................................................
Murray Altheim <murray18 at altheim dot com> = = ===
http://www.altheim.com/murray/ === ===
= = ===
In the evening
The rice leaves in the garden
Rustle in the autumn wind
That blows through my reed hut.
-- Minamoto no Tsunenobu
_______________________________________________
DPRGlist mailing list
DPRGlist at lists.dprg.org
http://lists.dprg.org/listinfo.cgi/dprglist-dprg.org
More information about the DPRGlist
mailing list