If we remove a connection with someone, will the other person get to know?

No, they won’t. But they can see that you viewed their profile since you had to visit their profile to remove the connection.