[Dprglist] PID-tuned Clock in Python?
Chris N
netterchris at gmail.com
Sat Feb 20 09:43:06 PST 2021
Lot’s of good advise in here!
The recent Cortex-A + Cortex-M combo SOCs are certainly interesting. Hopefully the boards come down in price a little. I have seen one from NXP with a i.MX8 + Cortex M7. I.e. Quad-core Cortex-A53 at 1.5Ghz, i.e. same caliber as Pi but with built-in Teensy 4.0-caliber coprocessor.
How is this different from connecting a STM32G4xx Nucleo board or a Teensy to the Pi? I think the main advantage of having the Cortex-M4 (or M7 in some cases) as a co-processor on the same silicon is that you get a low-latency high-bandwidth connection between the two, compared to I2C, SPI or UART. Probably (hopefully) there are libraries available to take full advantage of this internal interconnect.
What it DOESN’T get you automagically is rugged, 5V-tolerant I/O that is exposed via a prototyping-friendly connector that carries Signal+GND+PWR (that last bit is of course a board-level thing, not chip level)
Murray – could you please state what exactly the timing requirements of your control logic are? What would you consider “good enough”?
And what exactly happens when occasionally you don’t meet your requirement? I think Karim made some good points in that regard. Maybe you can just let it go? Maybe its one of those times where you are trying to improve or optimize something but it is actually already “good enough”?
Frankly I don’t understand all that talk about connecting an external device to the Pi for the purpose of generating a “timer interrupt”, or even reaching down into the low levels of the Pi hardware and tapping into the hardware timers. Linux does use the hardware timers internally.
Now if somebody is bit-banging something and needs a delay of exactly 2.567 microseconds, then one definitely needs to use the hardware timers directly and there are libraries available to do so. (but if one is bitbaning stuff on the Pi than one is probably using the wrong tool….)
I have said the following previously, so apologies for the repeat, but it is worth saying again:
For anybody who is using the Pi, and who has
• actual timing REQUIREMENTS or expectations expressed in single-digit milliseconds or smaller
• or who simply wants to make sure the Pi is firing on all cylinders (timing wise),
• or who simply wants to rule out timing issues as one of the variables while trying to get stuff to work:
If you are not doing ALL of the following, then you are simply wasting your time:
1) Run your application – and all its threads - at elevated priority. Not only that, be sure to run your real-time threads at higher priority than your non-real-time threads. The same would be true when one is using a RTOS. The OS can’t magically know which activity is most important. You have to tell it.
2) Use a real-time Linux kernel (keyword: PREEMPT_RT)
3) Tell the kernel to isolate one of the Pi’s 4 cores and then run your real-time stuff on that core (add isolcpus=3 to cmdline.txt)
1+2 are simply a must.
You are not allowed to have any expectations whatsoever regarding fully predictable timing if you are not doing 1+2.
Period.
3 is icing on the cake and will shave off only a few more microseconds in practice.
4) It is also implied of course that the real-time threads in your application don’t do things are incompatible with real-time. For example, perform complex file I/O or call complex library functions and expect those to finish their work in a predictable amount of time. The same would be true in case of a RTOS
“Linux is not a RTOS” ? I would argue, if you do 1+2+3 then it is.
Most recent test on my Pi4 with above 4 prerequisites met:
Background workload (I’m deliberately giving my setup a hard time):
• Picamera was streaming 30fps video to my cell phone via wifi
• USB-connected web cam was displaying on the Pi Desktop (using the VLC app )
• Pi Desktop was accessed via VNC, connected via ethernet.
• Talking to Arduino via USB-to-UART (i.e. the one built into the Arduino)
• Basically there were about 10000 interrupts per second firing in the background
Application: real-time benchmark application “cyclictest”, running at max priority, taking 1000 samples per second, basically doing a sleep(1000us)
After 20min or so and 1M+ samples later:
• On average, a sleep() of 1000 microseconds came back within 1011 microseconds, i.e. error of 11us
• 90% of the time, sleep(1000us) came back in less than 1015 microseconds
• 99.9% of the time, sleep(1000us) came back in less than 1050 microseconds
• 99.999% of the time, sleep(1000us) came back in less than 1100 microseconds (i.e. error of 100us)
• Worst case observed during this 20min test: 158us
In other words, come hell or high water, the scheduling latency will be < 100us.
Throw in Covid-19 and the current conditions in Texas, let’s just say true worst case is somewhere around 200us.
If that is not good enough, then follow the advice given previously about normalizing / compensating.
Could somebody please show me a use case where the performance I just listed, coupled if necessary with normalizing the data, is not good enough?
Now if your use case is bitbaning a software UART at 115200 baud, then you are simply using the wrong tool, so it doesn’t count.
Now regarding Murray’s situation – throw in python with multiple threads implemented via the python “threading” module, then critical prerequisite #1 is no longer met, because python threads don’t have priorities or preemption.
Murray – I would still encourage you to measure your timing errors after meeting prerequisite 1+2+3. I always use a Pi4, because frankly the extra performance of it has a positive impact on timing, but I have recently built a real-time kernel that is compatible with a Pi 3. Once I have done a quick sanity check on it, I will make it available for download.
Chris.
From: Murray Altheim via DPRGlist
Sent: Saturday, February 20, 2021 5:35 AM
To: dprglist at lists.dprg.org
Subject: Re: [Dprglist] PID-tuned Clock in Python?
On 20/02/21 4:25 pm, David Anderson via DPRGlist wrote:
> Howdy,
>
> Well now that Karim and I are officially ensconced in the realm of
> oldfartdom, it occurs to me that this is true of most of the DPRG.
> Sad but true. And we would have gotten away with it too, if it
> hadn't been for you nosy kids. And get off my lawn (been waiting
> my whole life to say that).
Heh. Isn't your Texas lawn all covered in snow right now?
As you may be aware I'm not so far behind you towards that "get
off my lawn!" moment...
Actually, when I first moved into my house some neighbour kids
threw mandarin oranges at my front door and put a lemon in my
mailbox (not like the eggs and toilet paper used in Iowa) and I
chased them in the dark for several blocks, for sport. I heard
later from the mom (a local policewoman) of two of the boys that
they'd talked about it for days, I'd gained some respect among
them, and she totally thought the chase was a good idea. Those
kids are now all six foot tall teenagers and could likely beat
the hell out of me now. Nice kids.
> And on that note:
>
> On 2/19/21 6:49 PM, Murray Altheim via DPRGlist wrote:
>> A Pi is not a microcontroller,
>
> The Pi most certainly is a micro controller. What software is
> run on the hardware does not change the nature of the device.
> I'm not sure where you got this idea, but I've heard you repeat
> it before. Perhaps you believe Linux is the difference.
Well, I'm certainly aware that the 64 bit, 1.4GHz four-core
Cortex-A53 with 1GB of memory on my Raspberry Pi 3 B+ is not
fundamentally different in *nature* to the 32 bit, 120MHz single-
core Cortex M4 with 704KB of memory on my Itsy Bitsy M4. The
latter is still significantly more powerful than a "classic
microcontroller" such as an Arduino Uno running its 8 bit 16MHz
(single-core of course) ATmega328P, with its 32KB of memory.
So is there a difference in *nature* between a microcomputer and a
microcontroller, one that makes a Raspberry Pi the former rather
than the latter? What defines such a difference, if there is one?
As you correctly note, it's certainly not the CPU: that seems to
be more a matter of performance.
Is it the presence of the Linux OS, or is this (as you suggest)
something still potentially common between the two? Is this a
difference simply of colloquial, informal terminology? Is this
a slippery slope or is there some chalk line that is crossed that
makes a board a microcomputer?
It's easy to notice that microcontrollers typically have (or at
least can have) significantly lower performance specifications
than microcomputers. I believe this is the indication of the
difference in nature between the two terms, since it's clearly
not the *nature* of the CPU itself.
The difference lies in the features that are capable of being
supported on a microcomputer board by that extra power, and the
presence of those features (not the power of the CPU) is what
constitutes the difference in nature.
The difference that makes a Raspberry Pi a microcomputer -- i.e.,
more than a microcontroller -- is the much larger set of features.
Yes, the necessary presence of the time-sharing Linux OS as a
more sophisticated, complex OS capable of a supporting a user
interface, multiple users, Bluetooth, WiFi, an HDMI monitor (or
two, with the Pi 4).
As what became fairly clear with the Raspberry Pi 4 and now with
the new Pi 400 Desktop, you can plug a keyboard, a mouse, and up
to two HD HDMI 1920x1080 monitors into it and you have a full-
blown personal computer with a full desktop environment including
a web browser, Open Office, a high quality music player, etc., a
personal computer with a lot more power than what we called a "PC"
in the 90s. E.g., a NextStation in 1993 by comparison was running
a 32 bit, 25-33MHz Motorola 68040 with 8MB of RAM, supporting a
single 1120x832 4-color (B&W plus 2 grays) display.
So yes, at the higher-performance end of the microcontroller range
you may find an embedded Linux operating systems, but that still
doesn't make them the basis of a personal desktop computer. Any
of the more powerful Raspberry Pis can be. It's that having that
amount of CPU power provides the ability to make it a microcomputer.
> The reason Linux is not run on the STM32 Nucleo family is not
> because they are "micro controllers." But rather it is because
> of a lack of memory; not because they are basically some different
> sort of beast. They are not. In fact, Embedded Linux DOES run on
> the STM boards ... see eLinux.org. But running Linux or embedded
> Linux on the device does not change it's nature. It is still a
> micro controller. And so is the Pi, with or without Linux.
I think we're kinda agreeing here except for the last sentence.
The STM32 is a microcontroller board. It's performance-wise not
capable of supporting all the features that are be expected of a
microcomputer, such as multiple users, WiFi, BlueTooth, a full
desktop user interface, etc. But a Pi can do those things, which
puts it into the category of a microcomputer. More powerful than
a NextStation. The presence of Linux on an STM board isn't there
to support a desktop environment, it's there for embedded usage,
as indicated by the name.
And it's true that this rather artificial boundary is sometimes
a bit blurred, as there are some very powerful CPUs being used
on boards that are designed as microcontrollers, boards without
all the various bells and whistles (like HDMI and stereo audio)
that might lend itself to a desktop user interface and being
used as a microcomputer.
So it seems that maybe the real difference is not one of CPU
type (clearly) but one of board design and application. This
doesn't mean a microcontroller is a lesser beast, it's just
built for different purposes and therefore has different
features and IO. As you have pointed out, a microcontroller,
due to its focused design, is better for RTOS than the Pi,
which might be the same CPU architecture under the hood but
is weighed down by the baggage necessary to provide a very
different application-level support.
I happen to be using a Pi on a robot, so maybe I'm doing it
all wrong. I prefer to think that my requirements are different.
You've done absolutely amazing things with the STM32. Nobody
could ever say it's not fit-for-purpose.
> Along the same lines, I think you are using a Rasberry Pi2 or Pi3?
Both of my robots are using the Raspberry Pi 3 B+ with 1GB RAM.
> Those devices use a quad core Arm Cortex-A53. That means there are
> actually FOUR sets of hardware timers, one associated with each core.
> The generic timers so provided have 32 bit and 64 bit modes and are
> integrated into the ARM Nested Vectored Interrupt Controller. The
> ones in Core0 is the only ones used by the Linux implementation.
> I believe they provides a 1 MHz clock. The others are available
> without constraints to the user. That's you.
Aha! (!) Now we're getting somewhere, and somewhere that might lead
somewhere good. This is why I like being in this group, and why I
really appreciate you old fart nitpickers.
> I'm not sure it has occurred to you that the timing signal you are
> currently deriving from an external ARM Teensy and then read by an
> interrupt on a Pi GPIO pin can instead more simply be generated
> internally from one of those hardware timers and interrupts on the
> Pi itself. No external devices or GPIO pins required. Yes the Pi
> ARM processor does have a nested vectored interrupt controller.
> It's very cool.
Now I hardly claim to be any kind of expert either on the Raspberry Pi,
or on Python. So no, it hadn't occurred to me by a hardware timer on
the Pi, mostly because of the libraries I've used I've never seen any
support or mention of this. If you search on "Raspberry Pi interrupt"
you'll find a whole lot of stuff on the fake-interrupt/polling-GPIO-pin
within the two most popular Python libraries, and so far I've seen
nothing on using those hardware timers directly. That's not to say
there isn't something in Python, I just haven't seen it. [anyone
reading this far down this rabbit-hole thread have any idea?]
> Perhaps this is not possible with python, I don't know. But it's
> certainly possible in C/C++, the language in which Linux is written.
> (Python itself is written in C, I believe the base is actually
> called CPython.)
CPython is the reference implementation of Python, written in C.
There's also PyPy, a faster, just-in-time (JIT) compiler, faster
because CPython is an interpreter. On the Pi I'm using CPython.
> So while it might not be possible for you to access those timers
> from python, that is a limit of the chosen language, not the Pi
> hardware.
Well, my guess is that if it's actually possible someone has already
done it on a Raspberry PI, at least in C/C++, and if it can be done in
C one can either wrap those C calls in Python (to use within a
Python-based ROS), or go to the trouble of creating a Python library
that does the same thing. I'm simply unaware of such an existing
library, and I've not done much research into programming in C/C++ on
a Pi since one of my robotics goals has been to learn Python.
> So thinking that "A Pi is not a microcontroller" is not only untrue
> but also probably limits one's understanding of what the device is capable.
We might quibble about the what constitutes a microcomputer and
what constitutes a microcontroller, but this conversation has
brought out a more interesting topic for anyone who might be
contemplating similar timing issues on a Pi, namely that there
is a real possibility of figuring out how to use its hardware
timers rather than software timers that are affected by system
load. I may be entirely ignorant about this -- maybe lots of
people know how to do it. Not me.
I will assume from your comments that the unused hardware timers
on the Pi wouldn't be adversely affected by a big load on the
core running my Python application... it doesn't seem to make
sense that they would.
> Now if one were to rephrase that as "Linux is not an RTOS" (but
> both run on micro controllers) then you get no argument from me.
> Like "Photoshop is not a MySQL Database" (but both are software)
> or "Cheese is not fried chicken" (but both are food).
I believe you're using the term "micro controller" here to refer to
the CPU itself, which we agree are both ARM architecture Cortex-Mx,
but the PI itself is more than a microcontroller, not because of
Linux but because of its different nature as a board. Not because
of the CPU.
I'm really, really trying not to quibble over terminology, except
insofar as doing so gets us closer to a better understanding. I've
been too long in the IT world to want to ever get into another
argument over what is a better programming language, operating
system, brand of computer, whatever. (whatever!) Life is too short.
But again, we are very much in agreement that the block I've had is
due to me *not* thinking of the microcontroller nature of the Pi
underneath its board-life as a microcomputer. I've been treating it
as a microcomputer and not investigating its hardware more. My bad.
> So just a hint, and I'll retire from the field. You currently have
> an ARM Teensy
[Actually an Itsy Bitsy M4, more powerful but same basic CPU as the
Teensy.]
> generating a timing pulse to try to get the Linux/Python on the Pi
> to do something that it is not suited to do, which is precise real
> time timing. The somewhat Rube Goldberg-esque "rabbit hole" to which
> Karim refers. Agreed. I'm happy to climb out of that hole if I can find a better
solution. It was all I could come up with. It works pretty well but
it is very Rube Goldberg-esque.
> But you also have the Teensy, which is ideally suited to do those
> real time tasks, sitting there basically unused. A lonely micro
> controller crying out in the wilderness, "Murray! Murray! Use me!
> I can do the real time stuff and report the results to the Pi!
> I've got GPIO pins, timers, interrupts! And I'm already here!
> Turn me loose!"
Ahh, but here's the rub: I've already tried that, about nine months
ago. I'd tried the Arduino Firmata library and even went out and
bought an Arduino Nano 33 only to find out nobody bothered to
implement the Wire.c library's support required for Firmata, so I
went back to the Itsy Bitsy and ended up writing my own simpler
protocol to communicate between them. It was that whole Master Slave
thing I wrote up to connect to my Integrated Front Sensor (and got
a pleading email from some Czech guy hoping I'd give up the
master-slave terminology because he found it offensive).
I got that working but it never worked very well. It'd miss messages,
and was too slow. I tested it against the Nuvoton M51 IO Expander,
which proved about twice as fast in sending the same amount of info,
more reliable but that seems still rather on the slow side (though
I've realised I can optimise it somewhat).
In terms of protocols, there's I2C, serial, UART, none I've found
so far that satisfies me. One strange idea: because I'm doing most
of my controller and sensor coms over I2C I actually have 12 pins
free. That's enough to implement a kind of direct parallel communi-
cations between their respective GPIOs, basically three pins for
address, eight pins for data. Speed-wise it'd be very fast. Not
elegant, but very fast. Maybe a bit Rube Goldberg-esque.
The real problem has been (and remains) how to create that master-
plus multiple slave- or sub-processor architecture, and how to,
with very good performance, intercommunicate between not just two
CPUs but potentially many. Cars do it with CAN. I'm already doing
that amongst my rather large array of I2C sensors/devices.
I'm not sure how far I2C can on its own scale much further than what
I'm doing right now, and I'm ambitious: I've been communicating with
several people, doing research towards building a rocker-bogie or
triple-rocker robot. Six motors, encoders, and motor controllers,
either six or twelve servos, a lot more sensors, a lot more inter-
communication on the robot. Still Linux as the master (I still want
WiFi, ssh, the ability to do either mono or stereo vision), but I'm
looking for what we've been talking about: how to best implement
the architecture we've been circling around.
> So my advice, which I recognize is unsolicited (and perhaps > unwelcome): listen to that cute little fellow.
I hope you sincerely don't think the tenor of these messages indicates
anything you're written is unwelcome -- far from it. You've as I've
said before been very generous with your time with me, and I'm very
grateful.
I have tried to listen to that little fellow, but I think at this
point my interest has waned due to the limitations of the two-cans-
and-a-string communication, which I admit may be due to my own
limitations as a programmer.
----
I'm kinda interested in seeing if that STM32MP157A-DK1 board that
Kipton mentioned might prove more capable in terms of inter-
communications. Is the bridge between the systems fast? According
to Kipton's message it looks like it is. The board is compatible
with both Raspberry Pi and Arduino hardware accessories, so that's
a huge plus. While running a Cortex-A7 @ 800MHz and a Cortex-M4 @
209MHz the former is about half the horsepower of a Pi 3 B+, but
I'd (as you suggest) be offloading most the RTOS tasks to the
Cortex-M4 so that performance hit probably wouldn't be significant
to the performance of the robot, and certainly less Rube Goldberg.
On the other hand, if ST is able to (hopefully) seamlessly meld
these two processors on a single board, perhaps it's possible for
me to do better than I have so far using a Pi and the Itsy Bitsy.
But I'd have to figure out how to improve the performance of the
bridge, which has so far proven elusive (and frustrating).
And the STM32MP157A-DK1 is only (!) two processors. I might want
more than that for a six motor robot with encoders. Maybe a Pi and
six Cortex M4s.
But this conversation has brought out my curiosity (thank you!) to
investigate how I might use the hardware timers on the Pi.
I hope you're all staying warm.
Cheers,
Murray
...........................................................................
Murray Altheim <murray18 at altheim dot com> = = ===
http://www.altheim.com/murray/ === ===
= = ===
In the evening
The rice leaves in the garden
Rustle in the autumn wind
That blows through my reed hut.
-- Minamoto no Tsunenobu
_______________________________________________
DPRGlist mailing list
DPRGlist at lists.dprg.org
http://lists.dprg.org/listinfo.cgi/dprglist-dprg.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dprg.org/pipermail/dprglist-dprg.org/attachments/20210220/c61f09ce/attachment-0001.html>
More information about the DPRGlist
mailing list