In the keep, where you manage your characters, you cannot remove the third party member, you always have to have a minimum of 1 party member. It's not that big of a deal, but it's still annoying to have to find the party member you want, add him, tab back to the 3rd member you were not able to remove, remove him, and then go back to finding the other party members.
I agree that it's annoying - but if you use the store/retrieve party, you can store your different configurations through that. Just remember that it switches equipment if you don't resave it regularly.
Oh! Omg, that button it the bottom right-corner needs to have its little tutorial pointer at the beginning, or be placed closer to the character library sheets. This is exactly what I was hoping for (I was considering writing feedback about how it would be useful to be able to save a party, etc.) Thanks for the pointer!
It just got added a few days ago (we've been pining for it), I think it's supposed to have a tutorial written - might not have gone in yet.