No, removing from party is safe. There is a separate delete button that deletes characters, so as long as you don't press that - they're safely put into your left most pile in the keep.