My recommendation isn’t to ditch Signal for contacts that have it to, it’s simply to avoid adding another app with access to your SMS if you are worried about security, there is nothing that app will do better than your default SMS app.