At some point or another you’re going to have to do it. Google is not going to stop caring about secure sites. Yes, the 301 redirect could have an impact on ranking – you could lose a tiny bit of Domain Authority, but this is likely to be only temporary.