[Dprglist] Visualizing data [was: PID]
Murray Altheim
murray18 at altheim.com
Sat Oct 3 12:30:10 PDT 2020
Hi Chris,
I've got a copy of Open Office Calc, and probably an old copy of Excel
somewhere on a prelapsarian laptop leaning against a wall in the basement,
but what I'd like to do is, following a run (not during it), the python
script would grab the log file and POST it to an existing RESTful endpoint,
updating the visualisation. So there'd be no manual process at all, i.e.,
I'd do a run, then go to Grafana and see what happened. If I overwrite the
ID of the database record in Elasticsearch I won't even have to change the
URL, just watch the visualisation change.
This is possible with GnuPlot with a bit more manual labour, as I'd have
to do an ftp put onto my workstation and then run some kind of local script
to convert it to a graph, or something akin. I was at one point doing this
on the Pi itself but that means I need to have an HDMI monitor connected,
which obviously doesn't work except when the robot is sitting immobile on
the bench. Good for tuning an unloaded PID but not on the floor.
I can kinda work around anything, but the post-run posting of data to a
service that can visualise is a nice option, especially since learning
how to do that is also a good skill for my day job.
Cheers,
Murray
On 4/10/20 2:23 am, Christian M Netter wrote:
> Why not just use MS Excel or one of the free alternatives? Or are you trying to graph the data in real time while the robot is running?
>
> Chris
>
> On Oct 3, 2020, at 5:40 AM, Murray Altheim via DPRGlist <dprglist at lists.dprg.org> wrote:
>
> Hi Bob,
>
> As you may imagine, time to do things always seems too short. I had a
> go at installing both Grafana and Elasticsearch via Docker containers
> onto a "spare" NVidia Xavier NX I'd bought and unfortunately haven't
> been able to use for its intended purpose [I've tried 4-5 times to
> install the Cuda nvGRAPH library but the whole thing then fails to
> boot and I've just given up on this $800 or so of investment, a real
> regret, but that's another story...].
>
> I was able to create Grafana and Elasticsearch accounts, etc. but
> haven't managed to figure out how to tie them together with the ES as
> a data source yet. This involves creating a data source and pointing
> Grafana at Elasticsearch, but I haven't got that far.
>
> My Logger class extends the Python logging library, and I just made a
> minor change to now write just the PID data to a CSV compatible file.
> My new data output is as per David, Karim and others' suggestions
> (thank you!) now running on its own thread an entirely separate 20Hz
> loop (i.e., separate from the PID controller's own loops) that polls
> *all* of the data and writes it onto one line of the log.
>
> I then attempted to use the Python-to-Elasticsearch library and wrote
> a test class but for some reason the ping() feature failed to connect
> to the server (via IP address), so I gave up and began working once
> again on the log output from my PID controllers. At least that's done,
> it even prints out a header and legend:
>
> name : description
> kp : proportional constant
> ki : integral constant
> kd : derivative constant
> p.cp : port proportial value
> p.ci : port integral value
> p.cd : port derivative value
> p.lpw : port last power
> p.cpw : port current motor power
> p.spwr : port set power
> p.cvel : port current velocity
> p.stpt : port velocity setpoint
> s.cp : starboard proportial value
> s.ci : starboard integral value
> s.cd : starboard derivative value
> s.lpw : starboard proportional value
> s.cpw : starboard integral value
> s.spw : starboard derivative value
> s.cvel : starboard current velocity
> s.stpt : starboard velocity setpoint
> p.stps : port encoder steps
> s.stps : starboard encoder steps
>
> https://service.robots.org.nz/wiki/attach/PIDController/ros-2020_10_02T20_31_53_278210.csv
>
> David Anderson seems to have a (as an outside observer) very quick process to
> get data into GnuPlot, or maybe just a ton of experience. I'd used GnuPlot on
> my v1 PID controller but would both as a hobbiest and software professional
> love to be able to populate an Elasticsearch DB and automatically get that
> into Grafana. I have no need for history so overwriting the ID of the DB
> record would be fine.
>
> If you've got some relatively easy way to get CSV into Elasticsearch I'd be
> quite happy to learn how.
>
> Cheers,
>
> Murray
>
>> On 3/10/20 1:46 pm, Bob Cook via DPRGlist wrote:
>> HI Murray,
>> Grafana does pretty slick graphs without much work at all. Here is an example image of a (not very good) PID implementation showing the target speed, actual speed, and error. As you can see this is going from speed=0 to speed=20 then speed=0 again after 30 seconds.
>> Happy to share the detail of my Docker setup if that would help. I’m using Docker Compose to create three containers: ElasticSearch, Kibana, and Grafana. The three containers share a private network, and the data for each resides on the host rather than inside the containers themselves - makes upgrading the containers easy without losing the data. Sort of a beautiful thing made simple by Docker.
>> - Bob
>>>> On Oct 1, 2020, at 1:24 AM, Murray Altheim via DPRGlist <dprglist at lists.dprg.org <mailto:dprglist at lists.dprg.org>> wrote:
>>>
>>> On 1/10/20 9:25 am, Bob Cook via DPRGlist wrote:
>>>> Hi Murray,
>>>> You may want to look at Grafana as a tool to visualize time series
>>>> data. It is open source and easy to set up if you are familiar with
>>>> Docker containers. It pulls data from a variety of sources, I’m using it with an Elasticsearch instance, also as a Docker container.
>>>> I set them up on a Linux host with Docker, but apparently you can
>>>> use Windows or macOS as your Docker host pretty easily too.
>>>
>>> Hi Bob,
>>>
>>> I've as part of various jobs designed and implemented both Solr and
>>> ElasticSearch services but I've never tried standing up a Docker
>>> instance nor have I used Grafana, though I've looked over the website
>>> a few times. I'm a Linux user so yeah, it doesn't sound like it'd be
>>> too difficult to attempt what you've done, maybe easier than trying
>>> to do it the hard way, which is kinda what I'd done on my version 1
>>> PID controller: write the data from the PID to a file on the robot,
>>> then either on a HDMI-connected monitor to the Raspberry Pi or on my
>>> workstation computer doing the GnuPlot visualisation. That took a
>>> fair bit of effort so if Grafana is substantially easier and more
>>> configurable I might give it a try. Good learning experience both for
>>> my robotics as well as job skills too.
>>>
>>>> I’m using Python to push data into the ES instance. There is a Python
>>>> lib for easy access to ES. My robot records a binary stream of stats
>>>> that I convert and upload offline, after I’ve collected data from a test run.
>>>
>>> Wow, thanks for that clue -- Python to ElasticSearch would be a pretty
>>> huge simplification if the ElasticSearch to Grafana connection were
>>> similarly easy. But again, a good learning experience in either case.
>>>
>>> If I were doing a microservice architecture between powerful services
>>> on a fast network I might want to send all the log messages directly
>>> to ElasticSearch, but considering this is running on a Raspberry Pi
>>> over WiFi and I don't need the data to be live-streamed, I'll likely
>>> think about how to load a completed log (as CSV, as Chris suggested)
>>> after the fact.
>>>
>>> Thanks much,
>>>
>>> Murray
>>>
>>> ...........................................................................
>>> Murray Altheim <murray18 at altheim dot com> = = ===
>>> http://www.altheim.com/murray/ === ===
>>> = = ===
>>> In the evening
>>> The rice leaves in the garden
>>> Rustle in the autumn wind
>>> That blows through my reed hut.
>>> -- Minamoto no Tsunenobu
>>>
>>> _______________________________________________
>>> DPRGlist mailing list
>>> DPRGlist at lists.dprg.org <mailto:DPRGlist at lists.dprg.org>
>>> http://lists.dprg.org/listinfo.cgi/dprglist-dprg.org
>> _______________________________________________
>> DPRGlist mailing list
>> DPRGlist at lists.dprg.org
>> http://lists.dprg.org/listinfo.cgi/dprglist-dprg.org
>
--
...........................................................................
Murray Altheim <murray18 at altheim dot com> = = ===
http://www.altheim.com/murray/ === ===
= = ===
In the evening
The rice leaves in the garden
Rustle in the autumn wind
That blows through my reed hut.
-- Minamoto no Tsunenobu
More information about the DPRGlist
mailing list