That is actually pretty much what happened.

The language they speak in Madagascar is a Polynesian language. Polynesia is north of Australia. Apparently the people in mainland Africa had not come to sail far enough to the sea in order to notice there’s a big-ass island to inhabit. Then they got visitors going “hi, wanna join us on our new place?”

Since nobody knows how the Madagascarians came there, the arrow is drawn the way it is.