[Dprglist] Timing in Python on the RPi

[Murray Altheim]

On 4/12/20 8:57 am, Chris N wrote:
> Murray – this is a follow-up to our discussion about timing, in Python,
> on the Pi during the 12/1 RBNV meeting.

Hi Chris,

I appreciate you digging into this. I have as well, with somewhat similar
results.

> (I can get into more details and/or show the tests in action on the
> command line during the next RBNV if there is interest)
> 
> I ran a few tests on my Pi 4 to see if there is anything unique or 
> different about Python when it comes to real-time performance. I 
> mainly wanted to check if Python performs any worse than a normal
> process.   (I don’t believe there is a way for it to do any better 
> than other processes….).
> 
> You mentioned time.monotonic() on the call, so I played with that 
> a bit but in the end I used time.perf_counter().  On the Pi the two
> timing methods in Python seem to be equivalent, but on Windows  > time.perf_counter() delivers sub-millisecond granularity whereas
> time.monotonic() gives you a granularity of 16ms or so.

As I don't use Windows (at all) that 16ms is not an issue.

My understanding of the monotonic clock was not quite correct. The monotonic
is used for *relative* (elapsed) measurements only. Its clock has a tick
rate of only 64 ticks per second, so while it is accurate over a longer
range, it's not suitable for short, accurate timings. But it is recommended
when measuring time *across* processes as it is tied to the system time and
therefore only skewed by clock drift, which on most systems is measured in
seconds per month or seconds per year (i.e., ignorable).

The perf_counter() has a much higher resolution so it may be considered
more suitable, but it is only accurate when called *within* the same
process, so basically useless in a multi-thread application.

> I usually use a utility called “cyclictest” (apt-get install rt-tests)
> to evaluate real-time performance on a linux system and a tool called
> “stress” (apt-get install stress) to create a load.     Cyclictest 
> basically does a sleep() and then checks if the thread woke up at the
> expected time.

I modified one of the functions I'd found as below:

from datetime import datetime
import time

def check_sleep(amount):
     start = datetime.now()
     time.sleep(amount)
     end = datetime.now()
     delta = end-start
     return delta.seconds + delta.microseconds/1000000.

error = sum(abs(check_sleep(0.050)-0.050) for i in range(100))*10
print("-- average error is {:0.3f}ms".format(error))

which is pretty similar to what you'd provided.

> It turns out that it only takes a few lines of Python code to replicate
> the essence of what cyclictest does, i.e.
> 
> Loop:
[...]
> Conclusion:  Python is bound by the same scheduling laws and limitations
> that other processes under Linux are bound to,  which means the following:
[summary...]

This is as I had understood as well, that Python-on-Linux is the issue, not
it being a Raspberry Pi or a Ubuntu desktop workstation with an Intel i7.

[...]
> If you want to do better than that, you need a kernel that has the real
>-time patches applied and configuration option PREEMPT_RT enabled. With
> this, the Linux kernel no longer enters these multi-millisecond long
> critical sections and so it doesn’t get much in the way of good real-time
> performance.   Also, the interrupt handlers of most device drivers now 
> get turned into prioritized threads and therefore even a poorly written
> device driver won’t get much in the way provided you use a priority that
> is higher than that of the interrupt threads.    PREEMPT_RT is required
> for ROS2 if you expect to benefit from the real-time improvements that
> ROS2 has made over ROS

I wonder how hard it is to actually modify the typical Raspberry Pi OS
(previously Raspian) for PREEMPT_RT? I found two references but haven't
tried this out yet.

   https://lemariva.com/blog/2019/09/raspberry-pi-4b-preempt-rt-kernel-419y-performance-test
   http://robskelly.com/2020/10/14/raspberry-pi-4-with-64-bit-os-and-preempt_rt/

> With this, latencies of <250microseconds even under all types of stress
> conditions are quite doable.  I have seen reports of <100 microseconds also.

Which of course would be fine.

I've also been looking into installing a RTC board on the robot, one of
those with the little battery, but it looks like while they sync the Pi's
system clock accurately, I don't think it fundamentally changes the issue
with millisecond-level accuracy, as would the PREEMPT_RT system patch.

I think part of the reason I've been able to ignore this kind of issue to
some extent is that, as David Anderson pointed out recently, my KR01 robot
is a Pi with a bunch of I2C devices, even for the motor controller, so I'm
not relying on real-time performance as I might if I were controlling the
motors or sonar via PWM directly connected to a microcontroller.

I think it was Doug Paradis that was suggesting that the I2C bus wasn't all
that reliable for inter-process communications, but I've again not found
that to be a problem (I believe). I've seen some drunken behaviour but I
think that's more likely due to my acknowledged misunderstanding of Python
threading, i.e, I was using a lot of threads and possibly starving some of
the ones I was using for decision making, though it's also possible that
the Gamepad device I was using to manually control it (over Bluetooth)
might have been contributing. There's a lot of variables at play.

I've been making strides in reducing both my use of separate threaded
processes and moving more towards my processes reacting to clock ticks
over a message bus, my latest "experiment". We'll see how that goes...

I am somewhat intrigued by the notion of using a suitable microcontroller
like that ST ones David A. uses, running something akin to the Python-based
"OS" I've built. I'd still want the Pi on board for ssh/TCP stack, and
being able to make quick modifications to the robot (over ssh), all that
convenience.

I don't want to be connecting a USB cable to the robot just to load the
latest OS version, that seems quite a step backwards. So I'm currently
investigating how I might have a Pi and a microcontroller onboard, using
the Pi over ssh to load the microcontroller's code. Not sure if that's
possible.... Anyone know how? (It'd be mightily convenient)

Cheers,

Murray

...........................................................................
Murray Altheim <murray18 at altheim dot com>                       = =  ===
http://www.altheim.com/murray/                                     ===  ===
                                                                    = =  ===
     In the evening
     The rice leaves in the garden
     Rustle in the autumn wind
     That blows through my reed hut.
            -- Minamoto no Tsunenobu