To be far I do that at the red rocket, so you aren’t alone in that regard.

If you take the fusion core out of the armor, the settlers won't be able to enter it.

I put all my power armor on the roof of the red rocket with stairs that go up and needs a jump and for safety reasons a take the cores out

I find it easier to just suit up and go

I also have like... 5 X-02 armor sets, 1 jetpack T-45, and 1 X-01

Kept a whole barn full of various power armors. Left out the power cores, and the settlers never bothered them. Probably safer that way for a few 'particular' folks.