Confidence, self-esteem, shyness: where do you start?

Most often, it is esteem speaking. Refusing requires believing your needs weigh as much as the other person's, and when you doubt your own worth, that equation does not hold.
Confidence comes in afterwards, at the point of acting: daring to voice the refusal, holding the silence that follows. That is why we often work on both, but in that order.

Both work, and that is good news: you can come in through whichever door suits you. Waiting to "feel ready" before daring usually means never daring at all. Whereas every refusal you own, every time you speak up and hold your ground, becomes evidence that feeds your confidence.
Conversely, if the judgment you hold about yourself is very harsh, starting at the root avoids pushing on a fragile foundation.

By understanding that a third position exists. Between fading into the background (saying yes while meaning no) and steamrolling (raising your voice, overpowering others), assertiveness means expressing what you think and need without crushing the other person or yourself. It is not a personality trait: it is a skill, and it is learned — phrasing, posture, the ability to hold a silence…
Our dedicated page covers this triangle in detail: Shyness & assertiveness.

You are paying the delayed price of a refusal never voiced. The forced yes brings relief in the moment, no tension, no conflict, then the frustration builds up, and it comes out another way: as irritation, as fatigue, or by doing things poorly, late, with a sigh. It is a sign that a boundary needs to be set upfront, clearly, rather than paid for after the fact.

It is possible: a difficulty that shifts depending on the context points to confidence, not esteem, you know how, but certain relationships reactivate old roles.
Many people revert, in front of their parents, to who they were at fifteen. Those are exactly the automatic patterns we work on defusing.

That is fear number one, and it is worth turning around: a relationship that cannot tolerate a single "no" is not a solid relationship, it is a one-sided arrangement.
In practice, setting limits clarifies far more than it breaks: the other person stops having to guess, and you stop accumulating resentment. People who know how to say no, incidentally, are the ones whose yes actually means something.

Yes, and from both directions. The body follows the inner state: a rising voice, hunched shoulders, rushed speech, so bringing that activation down already changes a great deal. But the reverse also works: a grounded posture and a slower pace send the brain a signal of safety.
For the side of this that happens in front of a group, our dedicated page covers it: Public speaking.

Four to eight for assertiveness, confidence, or a specific instance of public speaking. Sometimes three or four when the goal is precise, like daring to make a request or refuse a particular person. More, six to twelve, when self-esteem is what is at stake: that foundation runs deeper.
A useful marker: results are measured in your real life, not in session, which is why we agree on small experiments to try between appointments.