A yes is worth maybe 100 times the small ego hit of a no, so the math almost never justifies staying silent.
The idea
I was being interviewed for a book about friendship, and when we got to the skills that matter most, I blurted out: not being afraid of rejection. Or rather, knowing the risk of rejection is far outweighed by the reward of connection. A yes to an invite or an intro request is maybe 100 times more impactful than the small hit to my ego when someone can't make it or doesn't think the intro fits. I'm still thoughtful and targeted about who I ask.
How to do it
Find the outreach you've veered away from: the invite, the intro request, the one sitting in your drafts, the one you composed in your head and never typed.
Write what a yes would get you.
Write what a no actually costs. Usually it's silence or a polite no. Neither will kill you.
Ask: will the benefit of a "yes" make the risk of a "no" worthwhile?
I'll bet you... yes. Send it.
When someone does say no, count it as a blessing and turn your attention to the next person.
Example
Would a past client really be angry that you checked in? If a prospect declines your offer, is it that bad to get a clear signal and focus on everyone else? In my coaching we reminded real estate agents that almost 100,000 people in any given region are thinking of buying or selling a home. The job is to find the ones who want your help because of who you are, so anyone who won't work with you frees you up for them. And if you're waiting to feel confident before you hit send, I have bad news: confidence usually arrives after the send.
Fear-Setting: Spell out the real worst case of reaching out next to the best case. The risk-to-reward ratio is skewed strongly toward reaching out.
Most People Won't: Fear keeps most people from reaching out at all, so simply doing it puts you ahead. Their fear is your competitive advantage.
Divorce the Action From the Result: You control the outreach, never the reply or the result, so judge yourself on whether you reached out and how well you swung.
Choose Who You Work With: Most business problems start with choosing the wrong people to work with. A strong network gives you the luxury of choosing.