Men’s buttons are on the right to make it easy to do up with one hand for right-handed people.
When buttons were invented, only the wealthy wore them. Wealthy women had maids who would do the buttons up for them, and so the buttons are on the left so that they are easier for someone facing you to do up.
There was also another theory I’ve read about making it easier to undress each other, if that makes sense?
Your head’s in the right place. A lady was expected to be dressed and undressed by servants. But even a non-aristocratic woman would want to dress the same as the courtly ladies, and not wear men’s clothes so women’s’ shirts got buttons on the left side, regardless of the wearer’s class and the idea just persisted through to the present.
If you think about it, you’ll see how little sense it makes. Back before things like cars, motels, and telephones, having an affair was complicated. You couldn’t just pop over for a quickie. On the other hand, servants were cheap. Even a cop or a school teacher could afford at least one maid.