How to revive a mockingbird
Well, not a mockingbird, but a Blackbird. My Blackbird. And I am happy to report that it has been revived. How?
After discussing my problem on the libera.chat IRC channel #talos-workstation, I was assured by the helpful people there that the aspeed-i2c-bus irq error is completely normal and not the cause of my problem. The BMC reporting itself as NotReady, however, could very well be the issue, because when the BMC is not ready (or at least thinks it is not), the machine cannot be powered on.
I managed to obtain a complete systemd-journald service log from the BMC and shared it on the channel, where someone pointed me to this particular error:
download_manager.py[1089]: Traceback (most recent call last): download_manager.py[1089]: File "/usr/lib/python2.7/site.py", line 62, in <module> download_manager.py[1089]: import os download_manager.py[1089]: File "/usr/lib/python2.7/os.py", line 400, in <module> download_manager.py[1089]: import UserDict download_manager.py[1089]: File "/usr/lib/python2.7/UserDict.py", line 117, in <module> download_manager.py[1089]: _abcoll.MutableMapping.register(IterableUserDict) download_manager.py[1089]: AttributeError: 'module' object has no attribute 'MutableMapping'
Something was clearly wrong with the Python libraries in the BMC system. Since many of the services responsible for monitoring and controlling the machine are written in Python, this looked like a promising lead.
And indeed it was - ipl_status_observer.py and ipl_status_led_monitor.py crashed immediately after download_manager.py with exactly the same error. But how could anything be wrong with the BMC software when I had just reflashed it?
root@blackbird:~# mount
/dev/mtdblock4 on /run/initramfs/ro type squashfs (ro,relatime)
/dev/mtdblock5 on /run/initramfs/rw type jffs2 (rw,relatime)
cow on / type overlay (rw,relatime,lowerdir=run/initramfs/ro,upperdir=run/initramfs/rw/cow,
workdir=run/initramfs/rw/work)
This took me a while to figure out. The BMC uses an overlay filesystem composed of two layers: a read-only system containing the actual Python library files, and a read-write persistent storage layer. Among other things, Python stores compiled bytecode files (.pyc) in the writable layer. Somehow, one of these compiled files became corrupted. In the case of _abcoll.pyc, the file had shrunk to only a few hundred bytes:
root@blackbird:/usr/lib/python2.7# ls -l _a*.py* -rw-r--r-- 1 root root 18619 Feb 19 2020 _abcoll.py -rw-r--r-- 1 root root 379 Feb 19 2020 _abcoll.pyc
All it took was deleting the file and starting Python interpreter, which automatically recompiled the module:
root@blackbird:/usr/lib/python2.7# ls -l _a*.py* -rw-r--r-- 1 root root 18619 Feb 19 2020 _abcoll.py -rw-r--r-- 1 root root 25476 Jun 3 18:40 _abcoll.pyc
...and after a BMC reboot, that completed without any Python-related errors, obmcutil bmcstate finally reported the proper state:
CurrentBMCState : xyz.openbmc_project.State.BMC.BMCState.Ready
After that I entered the obmcutil chassison{/code] and {code}obmcutil poweron commands, which worked exactly as they should, and after two weeks of silence, the Blackbird finally powered on again.
I don't know - and most likely never will know - what caused the .pyc file to become corrupted. Still, I am glad the root cause turned out to be something so trivial, even if tracking it down took quite a bit of time.
The Blackbird has since received an AMD Radeon RX 580, is now fully operational, and I hope it will remain that way for at least another six years. There is simply no POWER/PowerPC alternative on the horizon, and other architectures are not really an alternative for me.
(June 8, 2026 at 12:00 PM CEST)
Black Hawk Down
...except it’s not a Black Hawk, but a Blackbird - my Blackbird.
(I have described my current desperate situation through various channels (mail, IRC) so many times over the last two weeks that I might as well turn it into a blog post. So here it is.)
Since October 2020, the Blackbird has been my primary desktop machine. Last year, when I bought a 4K monitor and paired it with a Radeon Pro Wx 2100, I eventually noticed that the card had some issues operating at this resolution in X.org with the amdgpu driver. When moving windows around, some of them would turn black (sometimes not the entire window, just diagonal triangular portions), and the same thing occasionally happened when switching virtual desktops.
To test the card in a more standard environment, I decided to build a dedicated PC box with the same driver and desktop environment to rule out any incompatibility on the POWER side of things. Therefore, I did not use the Blackbird much during the last two months; I only started it in headless mode once or twice to access my data.
The Wx 2100 behaved exactly the same in a generic PC, so about two weeks ago I bought another GPU and also a slightly larger case for the Blackbird so I could fit more hard drives into it. I moved the board from the old case into the new one, inserted the graphics card, and the BB did not start - and still does not to this day.
After connecting the power cable to the PSU, the front panel LEDs (top left corner of the BB board) go from what the user guide describes as Phase 1 boot (flashing in pairs) to Phase 2 (all flashing at the same time) and continue doing this forever (all green except the UID LED, which is blue). The LED near the ASpeed chip is also flashing. The ATX standby LED is orange, while the PSU ACT LED remains off.
I can log into the BMC over SSH, through the web interface, or via the internal serial BMC header, although the last option is somewhat problematic because the serial console is constantly flooded with the following messages:
[ 117.735371] aspeed-g5-pinctrl 1e6e2000.syscon:pinctrl: request pin 26 (F20) for 1e780000.gpio:306 [ 117.735414] Want SCU90[0x00000002]=0x1, got 0x0 from 0x063F0000 [ 117.735433] Want SCU8C[0x00000200]=0x1, got 0x0 from 0x000000FF [ 117.735443] Want SCU70[0x00200000]=0x1, got 0x0 from 0xF110D206 [ 117.735558] aspeed-i2c-bus 1e78a440.i2c-bus: irq handled != irq. expected 0x00001001, but was 0x00000001 [ 117.747444] aspeed-i2c-bus 1e78a440.i2c-bus: irq handled != irq. expected 0x00001010, but was 0x00000010 [ 117.758206] aspeed-i2c-bus 1e78a440.i2c-bus: irq handled != irq. expected 0x00001001, but was 0x00000001 [ 117.770092] aspeed-i2c-bus 1e78a440.i2c-bus: irq handled != irq. expected 0x00001010, but was 0x00000010
There is no reaction to the power button or to any attempt to start the Blackbird from the BMC (obmcutil poweron from the shell or the power button in the web interface). This is not surprising, because the user guide states that the BMC has to complete both boot phases before the machine can be started. And it never does, as confirmed by obmcutil bmcstate:
CurrentBMCState : xyz.openbmc_project.State.BMC.BMCState.NotReady
I tried to override the power management as described on RCS Wiki} by performing the ATX Force Enable with i2cset -y 12 0x33 0x01. All fans started spinning, the PSU ACT LED turned green, but nothing else happened, so I turned the PSU off again using i2cset -y 12 0x33 0x00.
I also inspected the board to check whether I had damaged it, but there is not a single scratch on it and, as far as I can tell, all SMD components are still in place. I have also already re-flashed OpenBMC and erased the GUARD partition to rule out the possibility that either of them became corrupted somehow.
But nothing changed - my Blackbird still does not start. If I had to guess, I would say that the problem lies within the BMC, which for some reason is not booting properly, even though I can log into it and use it. But I have no idea how to fix it, so I sent an e-mail to Raptor Computing Systems support and now I am waiting to see what they suggest.
Even in my current state of despair, I have no plans to abandon the POWER/PowerPC architecture, so I am already looking at other options in case my Blackbird never flies again.
But I still hope everything will be okay in the end. Because right now it is not okay, so it is not the end. (May 27, 2026 at 1:00 PM CEST)
Mac OS X Cheetah on Nintendo Wii
What a great time we’re living in! PowerPC machines have moved from “obsolete computers used only by people in denial of reality” to “interesting hardware for tinkering”. In recent years, that tinkering has brought us things like a port of Windows NT to the PowerMac or even to the Nintendo Wii. And now the Wii is catching up with the PowerMac, because in addition to Windows, it can now also run Mac OS X.
Right now it's only the very first desktop Mac OS X release — 10.0, a.k.a. Cheetah — but it’s still fair to say that Bryan Keller has done an incredible amount of work to turn this idea into reality. The console itself does use a PowerPC G3–based CPU, but that’s where any similarity to Apple hardware ends. There is no Open Firmware, no PCI bus, less (and split) RAM, and even the framebuffer uses a different color model than desktop Macs. Bryan bridged all of these gaps, wrote custom drivers where needed, and the result is a working Cheetah at 640×480 resolution, even with the Classic environment.
It’ll be interesting to see if newer versions of Mac OS X will follow. I have a Wii sitting somewhere in the closet, and since I don’t really have much time to play on it anyway, I might soon try to convince it to pretend it’s a PowerMac.
(April 9, 2026 at 2:00 PM CEST)
Mirari - A new PowerPC board in the making
I'm a bit surprised that none of the sources aggregated in the World section of this humble site have picked up on this yet, but a completely new PowerPC board is in the making – and what's even better, it's supposed to be quite affordable.
Known parameters:
- CPU: Freescale/NXP PowerPC T1042 (64bit)
- RAM: DDR3L (1.35V) SODIMM DDR-1600
- Storage: 2x SATA 2.0, 2x NVMe (second slot mutually exclusive with second SATA port), microSD
- Slots: PCIe x16 (four PCIe lanes used), PCIe x4 (one PCIe lane), PCIe x1
- Ports: 8x USB 2.0 (half internal, half external), 4x USB 3.2 Gen1 (again, half internal, half external), RS-232
- Networking: 10/100/1000 Mbit Ethernet
- Audio: 192kHz/24bit analog, S/PDIF / TOSLINK digital
- Form Factor: microATX, 24-pin ATX power, 4-pin ATX CPU connector
- TRION FPGA
As first reported by amiga-news.de, Harald “Geennaam” Kanning has developed a new PowerPC-based microATX motherboard called Mirari, built around the NXP T1042 CPU (four PowerPC e5500 cores). At the moment, five fully working units exist and, as you can see in the video below, they're already capable of booting and running Linux. In the future, the board is also expected to support MorphOS and AmigaOS, provided there’s enough interest and goodwill from the developers of those systems.
While the raw CPU performance is nowhere near that of Raptor’s POWER9 machines, it is on par with – or even better than – almost anything else currently available that's PowerPC compatible. And according to another member of the small development team, Dave “Skateman” Koelman, Mirari is expected to be by far the most affordable option of them all.
It's not clear yet what that exactly means, but since this isn’t a profit-driven venture – all involved have day jobs and just wanted to create hardware they themselves would buy if someone else had made it – we might be in for a very pleasant surprise. There's no official website at this point; all project communication and updates happen via the Amigans Discord server, which I joined in order to follow and report on the progress here.
Even though I already own one AmigaOS-capable PowerPC system (a Sam460LE), this feels like one of those “shut up and take my money” moments.
(May 12, 2025 at 4:00 PM CEST)
Large Language Models on ppc64le with Ollama
I've seen many IT hypes over the decades, and artificial intelligence - more precisely, large language models (LLMs) - seem to be the current one. Don't get me wrong, I think they are wonderful and useful tools. However, as with any other hype, their omnipresence right now is a little tiresome. But the question is, is it possible to run an LLM on ppc64le locally?
The answer is yes.
The open-source framework for building and running language models on a local machine, called Ollama, can be built and run on our beloved architecture. Nevertheless, there are some minor challenges along the way.
First of all, you need the most up-to-date version of the Go language compiler possible. Anything from the 1.23.x range should be okay, though my Devuan was still on 1.19, which was not enough. Fedora Server 41 has what Ollama needs, so I installed it in QEMU with KVM enabled and all my 32 threads available to the VM. Along with the Go compiler, you also need a C++ compiler, CMake, and Git.
After cloning the Git repository, building is quite simple and completes without any errors:
cmake -B build cmake --build build
Soon after that, however, a problem emerges when trying to run the result with go run . serve and Ollama quits with following error:
Error: inappropriate ioctl for device.
This error message seems a bit too generic. Yet, together with the keywords ollama and ppc64le, Google pointed me to a GitHub discussion thread where Lars Erik Jordet published two patches for ppc64le almost a year ago. The first fixes a problem during the build (which didn't occur on my setup), and the second one fixes the exact error I encountered when trying to run Ollama:
diff --git a/readline/term_linux.go b/readline/term_linux.go
index 2d6211d..69e05bf 100644
--- a/readline/term_linux.go
+++ b/readline/term_linux.go
@@ -5,10 +5,11 @@ package readline
import (
"syscall"
"unsafe"
+ "golang.org/x/sys/unix"
)
-const tcgets = 0x5401
-const tcsets = 0x5402
+const tcgets = unix.TCGETS
+const tcsets = unix.TCSETSF
func getTermios(fd int) (*Termios, error) {
termios := new(Termios)
Since this change affects only three lines in the file, I applied it manually, and from that point on, Ollama worked. As it runs in CPU-only mode (my GPU is too old to do anything except draw the screen itself), it's not exactly fast, but it's perfectly usable.
So even AI doesn't seem to be the proverbial nail in the coffin for ppc64le. We'll see what else the future brings to test us.
(This text was checked for typos and grammar by AI. Of course.) (February 4, 2025 at 4:00 PM CET)
You can find all blog posts in archive.