[Dprglist] Multi-threading on multi-cores

[Murray Altheim]

On 20/10/20 3:58 am, Chris N wrote:
> Murray,
> 
> My 2 cents  regarding
> 
> “/I wonder if anyone on the DPRG mailing list has any experience with the
> Orange Pi or NanoPi boards they'd like to share/”
> 
> “…/running a multi-threaded Python application on eight cores means you 
> get real multi-threading rather than simulated multi-threading/”
> 
> “…/a subsumption architecture for a robot OS in multi-threaded Python/”
>
>   * I think that an easy to use reusable Python-based framework for implementing
> subsumption could be something that other Python fans would want to see and use.
>  That said, I would not worry too much about multi-threaded vs not and
>     multi-core vs single-core.  Instead, your first priority should be the user
> experience, where the “user” in this context is the Developer, i.e. you and
> potentially others.    What APIs and abstractions should your framework provide, so
>     that your subsumption logic is easier to create, read and debug?

Hi Chris,

Agreed. As someone relatively new to really digging into Python (this is my first
substantial experience doing more than a simple script, I guess about a year now),
it's been a challenge. I've posted all my code so people could pick and choose the
bits they like, but to actually run up the whole package they'd need my hardware,
or something akin. The OS is, at a whole, reliant on using the same motor controllers
and I2C sensors as I've used.

But at its core the subsumption engine is fairly generic. How to abstract/extract
that from the rest of the OS is unclear, as my time is limited. If I was getting
paid to do this and maybe even had a team, it'd be relatively trivial I think.

The essential classes could theoretically be extracted out as a "subsumption engine"
and perhaps I'll do that as I begin working on a second robot that doesn't have the
same motor controller, or potentially the same PID controller (since the encoders
and other hardware would be different I think it'd need a rewrite).

But the core could be cut down to something like the following:

    arbitrator.py      message arbitrator
    behaviours_v2.py   collection of behaviours (needs API)
    config_loader.py   YAML configuration loader
    controller.py      controller
    event.py           enum of event types
    fsm.py             finite state machine
    logger.py          logging wrapper
    message.py         message container
    queue.py           the message queue
    rate.py            provides constant time loop
    ros.py             the robot OS driver class

Source:  https://github.com/ifurusato/ros/tree/master/lib

The ros.py class would need simplification or more likely a complete rewrite
for each robot, as that's where everything gets tied together. But totally agree,
if anyone else wanted to use this I'd have a bit of a chore to clean up, document,
and make more generic.

>   * I only have experience with the Raspberry Pi series of boards and BeagleBone
> Black.   That said, I think they are all equally (ill) suited for robotics. 
>  Sure, you get a ton of compute power and memory for very little money, but you
>     usually get almost none of the I/O you need.   The BeagleBone Blue is maybe
> a notable exception, but still only a “nice try” since its built-in DC motor 
> drivers are underpowered and the the A2D inputs only tolerate 1.8V.

I didn't expect this to end up as a defense of Raspberry Pi vs. alternatives like
non-Linux ARM or Arduino boards, but I'm happy to oblige the discussion.

So: Linux vs. no Linux, which . It's certainly possible to build a robot without
Linux. Indeed, I don't have any statistics handy, but looking at the offerings of
robot kits online I'd say most are non-Linux, running ARM boards, Arduino-compatibles,
micro:bit, etc.

So yes, you get a ton of compute power when you use a four core Cortex-A72 with 8GB
of memory running at 1.2GHz, say compared with a single core 16MHz Arduino or a
180MHz Teensy. An Arduino Uno has 14 digital IO pins, 6 PMW pins and 6 analog IO pins.
The Teensy 3.6 has 62 IO pins, of which 25 are analog, 2 12 bit DACs, 20 PMW outputs,
so I'm guessing when you compare a Pi you'd compare it with something like a Teensy.

But any PI with a normal 40 pin GPIO bus is hardly poor by comparison. Any of the
general purpose IO pins can be configured as either analog or digital IO, and there's
the usual array of SPI, I2C, PWM and serial ports. There's 28 general purpose pins,
and if you extract out from that the I2C and PWM ones there's still over 20.

So what does anyone need more than 20 IO pins for on a robot? I'm sure there's use
cases, but I'm not one of them. Looking at my KR01 robot I've got 11 general purpose
pins I'm not even using. That's because I'm using I2C for much of my intra-robot
communications, e.g., motor control, sensors, etc.

As our sensors get more powerful the main board is left for computation. This is
assuredly a different approach than "traditional" Arduino robots, for sure, but I
don't see a Raspberry Pi as wanting in terms of IO. If my robot had arms and I was
running out of IO pins I'd be more likely to farm out IO to sub-processors like an
Itsy Bitsy, Arduino, or now, a Nuvoton MS51, as is increasingly being used in boards
by Pimoroni:

    https://shop.pimoroni.com/products/io-expander

So I'm a bit curious as to what kind of IO is lacking. I've got (for myself) quite
a few analog and digital IO pins, I've plugged in a few I2C-based ADCs (which are
higher resolution than the ones on an Arduino. E.g., the Pimoroni ADS1015 has four
channels that can read +/- 24V at 12 bit resolution, and my CPU isn't involved in
getting that data. My ThunderBorg motor controller is likewise I2C.

So maybe I've just made different choices in implementation as I don't feel I'm
living on the poor side of town.

>   * Apart from the lack of built-in rugged I/O,  the power consumption of most
> of these boards is also not that great (maybe less of an issue on larger robots;
> but definitely an issue for bolting one onto a 3pi which is powered from 4xAAA).

Of course, running an eight core CPU at 1.4GHz is going to chew through 4 AAA
batteries. I run my KR01 on 18V 1.5, 4.0 and 6.0 Ah power tool batteries, suitable
to the load. In a quiescent state my KR01 will sit on the bench for hours and
hours, until I start running the motors. It'll be interesting to see what that
Fire3 is like by comparison. But yes, the Pi 3 B+, Pi 4, Orange Pi and NanoPi
I've ordered run up to 2A and 3A of current on their power supplies.

On my smaller robot running a Pi Zero W I run that on a 5V USB power supply, and
the size of those has come down very considerably over the past few years. I'd
thought about Li-Poly batteries, etc. but getting them in NZ is impossible as
they can't be shipped by air (too dangerous) and the prices in NZ are way too
high to justify. That said, the Makita batteries are very expensive, but I have
them around the house anyway since I have the power tools (and now a vacuum
cleaner) that use them.

But I'm getting compute power, a Linux OS with a full complement of normal Linux
and Python 3 code, one or two HDMI ports if I want to plug into a monitor or two,
and for me one of the more important considerations: WiFi and ssh. You can't do
that (at least easily) on an Arduino.

And what kind of running time is an expectation on a robot? I would say that these
kinds of considerations are part of anyone's design. If I wanted a robot to be able
to run for four hours I'd expect to need a battery that big. In my case I chose
the Makita power tool batteries because I can swap them out of the robot very
easily -- the battery is just sitting on top of the robot. It is, yes, annoying
that I have to shut down the robot to do that, but that's 25 seconds. I can deal
with a loss of 25 seconds of my life. Charging 4-8 LiIon or LiPoly batteries and
having to potentially remove them from the robot is equally annoying.

>   * In addition, the lack of “instant on” – “instant off” is annoying.  
> Yes it is possible to bring boot time on Linux way down, and I have
> experience in doing this, but you give up a ton of flexibility.

The instant-on of an Arduino is nice, sure. But I'm not really concerned with
boot time as it doesn't really impact anything. I'm not running my robots in
competition, so about 25 seconds of startup time is inconsequential. Much more
concerning in comparison with an Arduino is that there's a chance of corrupting
an SD card or drive if power is suddenly shut off. I've had that happen just
once about six months ago and it was a pain, but now that I've got all my code
up at github I can rebuild the OS in pretty short order, maybe 20 minutes or so
to re-load Linux and then my python code. Annoying, but for me the WiFi and ssh
easily makes up for that. *That* is what I'm paying for.

>   * Of course our hardware should be powerful enough so we don’t get bogged
> down by having to optimize code all the time, and so that we can afford to
> run Python and whatever else we need to be productive. But unless you are 
> into ROS, image processing, 360deg spinning Lidar or all of the above at the
> same time, you won’t be able to really take full advantage of more than 2 cores.

Well, I actually am taking advantage of those cores when running a heavily
multi-threaded Python application. I haven't counted but I'm probably running
at least a dozen threads, maybe more, so farming those out to four or more
cores is actually pretty sensible. Not strictly necessary, no. It operates
on a single core, albeit considerably slower.

>   * Do not make your application logic too dependent upon specifics of >     the underlying hardware platform – in this context: presence or
>     absence of multiple cores.

As I mentioned, from a Python programmer's perspective there's no difference
in running those dozen threads on a single core CPU than an eight core CPU
as Python itself is managing that. But on a multi-core CPU Python gets to
farm out the threads to those cores, so performance really jumps. And I'll
be running essentially the same Python OS on both my 4-8 core robot as on
a Raspberry Pi Zero W, which has only one core. But it's pretty clear even
in an ssh session on the Zero W that it's a 512MB single core, just in
response time. I don't think this will translate much to robot performance
as that robot doesn't have much by way of sensor or processor requirements,
no AI, no video camera, etc.  Indeed, as David Anderson has amply demonstrated,
one can run a pretty amazing robot on a single core ARM board, so I'll freely
admit I have more processing power than is necessary to run a robot like I'm
building. There's certainly room for AI-strength CPUs like the Nvidia Nano or
Xavier NX, but I'm not heading that direction. Now, if I was running AI-level
logic I'd certainly be relying on the 21 TOPS of processing power over 384
GPUs. But I'm more down at the small robot end of things.

So my application logic is entirely independent on hardware (I certainly agree
with that) but takes advantage of it when it's available.

>   * Of course you know this already, so just a reminder: Multiple cores and
> hence multiple threads actually running truly in parallel can get you into
> trouble quickly if you are not careful.    On a single core, any given 
> thread can make the assumption that no other lower-priority thread will
> “interrupt” the currently running thread.   With multiple cores, you can 
> have situations where lower-priority threads access the same data 
> _literally at the same time_, so more explicit synchronization is needed.

Yes, coming from being a Java infrastructure programmer for several decades
I understand the danger of a multi-threaded environment, and I've run into
deadlock situations as I've been learning Python, but generally speaking
my threads aren't interacting with shared data at all.

The basic architecture is a whole bunch of sensors that asynchronously fire
messages (containing an event type and potentially data) onto a message bus,
which feeds into a PriorityQueue-based "arbitrator" that grabs the highest
priority message (based on event type) every 50ms cycle and feeds that to
a controller. The controller dispatches commands to any number of behaviours,
be they lights, motors, etc. It's a basic publish-subscribe architecture,
where the threads are entirely internal to each entity in the system. I'm
not running parallel threads or pools or anything corporate-like.

In summary, I think part of the reason I'm on a Pi using Linux and ssh is
the development environment. I built an Arduino robot, where everything is
in one single file with a loop() function. Yeah, we've on the VTCs talked
about the pain of using the Arduino IDE. It at least has highlighting
and code validation. But to me what was most painful was everything-in-one
-file. Very limiting.

I completely understand the idea of code economy. I've programmed in
assembly in the past. And I completely appreciate the amazing things that
David and others have done on tiny microcontrollers. As he mentioned to
me, I'm "using an awfully big hammer for some pretty tiny nails", but I
rather appreciate being able to use Linux, vi and ssh as my environment.
It's not Java on Eclipse, but it's also not a single file in Arduino IDE.

There's an aesthetic and philosophical question here that isn't a better
or worse than question. There's no doubt I could do more with less, but
we all come to this field with different goals, different tool sets,
different approaches. I've written what I think of as a subsumption
architecture, but done it entirely different than David or Rodney Brooks,
using a much more complicated architecture (than was necessary, yes),
using an appropriately-sized CPU that can easily handle it. It was part
of my exploration of Python and robotics.

Cheers,

Murray

...........................................................................
Murray Altheim <murray18 at altheim dot com>                       = =  ===
http://www.altheim.com/murray/                                     ===  ===
                                                                    = =  ===
     In the evening
     The rice leaves in the garden
     Rustle in the autumn wind
     That blows through my reed hut.
            -- Minamoto no Tsunenobu