Short answer: phone manufactures hate you
Long(ish) answer: Each phone manufacture heavy modifies the linux android kernel in order to make it work on there hardware. This usually means that the binary blobs (there are a lot of them) are only compatible with specific kernels.
In addition, phone manufacturers write bad code, that would need to go through many review iterations until the open source community is happy with their changes. Often their changes to the kernel break things for other hardware, for instance.
Review and writing good software takes time, and phone manufactorers don’t care about long term support, they want to sell you a new phone every year, or even more often. So they fork the kernel and hack together with lots of string and duct tape, a frankenstein kernel that just works on their hardware, but is broken for anything else.
It took a lot of work to get PC parts to become interoperable. There’s all kinds of special negotiations that happen at boot to discover the state of hardware. There are standard drivers for most peripherals
Mobile devices aren’t expected to change hardware so everything is hard-coded into the ROM. No discovery protocols are supported. Standardized drivers make it hard to ship new features or squeeze every last bit of performance so no effort goes into making them
Because the nuance of calling a PC a PC is that it supports a standard motherboard configuration and any Linux kernel will be able to boot. Mobile devices aren’t like this and every component requires a specialized driver to work.
And just to add to that, anyone who has installed Linux before knows that everything isn’t guaranteed to just work out of the box. Laptop WiFi cards, for example, frequently have driver issues. So it’s not quite as simple as OP is saying, even with PCs.
Reminds me of my fun times with ndiswrapper and nvidia-bl