How does adding irrelevant connections cannibalize my target audience?

Irrelevant connections will most probably not engage with your contact. Hence, the algorithm will not send it to other people as well.