It is hard to say no to people. You worry they won't like you. Or that you are missing out on a good opportunity. That it will negatively impact your career somehow.
But any successful career requires balancing a lot of different objectives. You can't say yes to everything (here's a post on how to choose), but there are good ways to get to no.
For the first step of the system, planning and paying the setup costs, take some time to do your own research on best practices. There are many good books on how to say no that may help. But the best strategy may depend on your position, personality, gender (I know it shouldn't, but alas). So keep track of different strategies you employed and how effective they were. Here are some of my favorites.
1. Try to get to yes. If it's an event you'd like to do, but the conditions aren't right, give the organizers a chance to get you to yes. Be polite but direct. Tell them why you are inclined to decline and ask if different accommodations can be made. I've asked for everything. Plane tickets for my family when I was breastfeeding and needed to travel with the baby. Private tours of bases and museums. To give the talk but not submit a requested paper. Different dates, times. Formats. More money. Better travel accommodations. Whatever you need to make the event/conference/workshop worth a yes. If the organizers can accommodate, great! Now you can participate on terms that work for you. If they can't, then you've said no, but in a less direct way.
2. Make a list of what you say no to. Sometimes I need some motivation to say no. In the moment, it doesn't feel good. And then I forget what I said no to, so I don't have the positive feeling of all the time I saved for myself. So think about saying no as buying time for yourself. Not only for other work, but you know, for life too. Make a list. And then relish that you are outside doing yoga instead of on a call.
3. Take the time to respond appropriately. This may seem obvious, but no one that is interacting with me is too busy to take the time to email me back. I find it super annoying when I'm putting together an event to not get responses. I mean, really people. So I always make sure to respond in a timely manner (for me, it's 24 hours because I always clear my inbox, but I realize that's rare) and graciously. Use punctuation. And address it to the person. Sometimes when I've been super busy, like just had a baby, or I'm on military orders, I use a template for my responses or an automatic response of 'no.' But I still respond.
4. Ask them to contact you another time. This is more for meetings, but I get dozens of requests each week from people who want to meet with me. To figure out who really values my time and expertise, I often ask them for additional materials or contact me later. Many times, people don't follow up, and I'm off the hook without feeling bad. If they make an effort, then so do I.