IMO the gender part just became the term because female and male pronouns happen to be separated by many of these systems.
What if nouns were categorised in Welsh (a “gendered” language) by their feline (femme) or canine (masc) traits, or some other arbitrary distinction that was lost over centuries of linguistic shift to align with Anglo-Saxon sexual hierarchies?
It seems small, but subverting the idea of “binary gender” in languages is one of the ways we can give people the language to describe sexuality and gender as a spectrum.
Any linguist chads who know more about why we use “gender” and “masc vs femme” and what people are saying about this in a world where that binary isnt useful anymore?

Hittite’s noun classes were for animate and inanimate. This might be what Proto-Indo-European (or before) originally used.
Both Greek and Sanskrit grammarians discuss grammatical gender using “gender” terminology. The author of the “earliest surviving medieval treatise on Greek syntax” was probably synthesizing earlier scholars rather than introducing new terms when he wrote in the early 800s CE. Greek actually has five genders - masculine, feminine, neuter, common, and epicene. Common is for nouns that are “capable of being either masculine or feminine,” and epicene is for “a word with a fixed gender used for both masculine and feminine beings.”
The Sanskrit word for gender, “linga,” appears in Pāṇini’s grammar, which may have been written in the 4th century BCE.
Sources:
Ancient Greek Scholarship, by Eleanor Dickey, p. 127, 167-68
Panini