I'm just finishing up something like a dozen shows nearly back-to-back. All of them were musicals, and on most of them I was the sole designer. Most were also children's or youth theater (which brings an additional set of constraints) and several were on extremely short tech schedules (which also introduces certain constraints).
When you are designing sound for musical theater, you are really doing three different (but intersecting) designs. You are designing sound effects (which for a straight play, would be the only thing you did), designing vocal reinforcement of the singers/actors (which for some musicals, is the only thing you do), and designing the reinforcement of the orchestra (which is basically the same job as doing sound for a live band).
Taking the last first, I work at a couple of rental facilities where it is not uncommon to have a band come in without their own sound guy. So as the house tech, I figure out how best to mic them up, work up some plausible settings, then mix them from the console during the performance. This you may be doing for stage musicals as well -- but for the musical, your primary requirement is to make it work for the combined performance. (Often as not, that means asking the band to turn down instead of mic'ing them to get them up.)
First step is learning what you've got to work with. Is it a small space and there are few or no wireless microphones for the cast? Then the task is control of the band -- getting them into a pit or behind a wall if anyone lets you do that! Is it a medium-sized house, the actors are on mic, but the band is tucked behind scenery far upstage? Then you may need to mic them, not for sheer volume, but for presence and intelligibility.
Often as not the first, last, and only thing you'll do is stick the piano into foldback monitors so the cast can hear the music while they are singing. You want this in place as soon as possible -- they'll be wanting it for rehearsals as well. Since we're not recording a concert grand playing Rachmaninoff here, an SM57 stuck in the lid of an upright, or (my new favorite) an SM/PG81 aimed at the soundboard from about six inches away is fine. Often the keyboard will be electronic. Here you will want a DI (Direct Box) for clean signal to the sound board. In many of the cheaper keyboards, though, there isn't even an audio out. And if you plug in to the headphone jack, it will cut off the onboard speakers and you'll have to drag out a monitor so the keyboard player can hear themselves again. It is a simple and stupid trick, but aim a small condenser mic at one of the speakers from a few inches away, looking straight into the speaker. It's good enough to send to foldback monitors.
Also in re that, try not to aim the monitors at the audience. The best monitor is front fill. I have a pair of FBT speakers (Jolly 5RA) that have a really tiny footprint and a really wide pattern. I also use (in bigger houses) a pair of Yamaha MSR-100's. I stick these on the front edge of the apron, or bolt them to the front edge of the stage even, so they are as much out of sight as possible and don't get in the way of dancing but cover the width of the stage.
Depending on the shape of the stage, you may need to bolster these with a second or even third set of speakers from the wings. I've used speakers aiming sideways from the wings, speakers sitting behind the proscenium arch pointing upstage and in at a 45 degree angle, and hanging overhead from the flies.
Walk the speakers. Get a keyboard player, or record some MIDI, or plug a laptop into the piano DI and play a CD and check your coverage. You want a nice smooth flat coverage across the prime playing areas, and as little as possible leakage into the audience. Also keep in mind that during the show there will be a whole chorus standing between the speakers and the rearmost singers -- this is where having additional coverage from sides and above can really help.
But on to the rest of the band. Again, you aren't recording. Most theaters are small enough that there will be a lot of direct sound already. In fact, in most cases the drums will be too loud already, and the bass "loud" enough to be heard from the back of the house as well. Plus, once you've turned up the monitors until the cast is happy, piano will also be quite loud. And if you've got a standard Broadway score and ten or more pieces in the pit, you'll already have enough brass to blast the socks off anyone in the front half of the audience.
Your aim is to achieve three things; to balance the orchestra with itself (aka make the soft instruments loud enough to seat correctly in the mix), to achieve flat coverage into the depth of the seating (aka to make sure the people in the back of the house hear the orchestra nearly as well as those in the front row), and to make the orchestra sound "good."
The latter is where the difference between loudness and presence comes in. I mic drums, for instance. And I put a DI on bases. Sure, you can hear the bass from the audience. But what you hear is undefined flabby mush. Left alone, the bass would sound like a truck passing outside; nothing but low frequency rumble. So you add to the existing low-frequency content with mid and high-frequency that defines and shapes the sound until it actually sounds like a string bass from out in the audience. That same remark is true of the drums.
There's a couple of different ways you can go about planning this. Me, I attend orchestra rehearsal/sitzprobe and take notes, and I try to have a chat with the conductor/music director to get an idea of where they are going. I find out where the band will be located. Usually I can get a pretty good guess of how much reinforcement will be needed, and at that point I can rough out a pit plot based on how many circuits I have available and what mics are in the building.
At one house I work at we usually have an orchestra of over twenty pieces, and they are often tucked away where they can't be heard as clearly. Unfortunately that same house has almost no microphone circuits. So there I've learned to mic by section. For "Into the Woods" I had a grand total of four microphones; a strings section (large diaphragm condenser positioned above the section), winds section (which also picked up a fair amount of brass), percussion overhead (mostly to boost vibes and marimba), and the piano mic.
Remember, your aim is to lift the soft instruments, and to bring presence to instruments that would otherwise sound muddy. You have to listen to the sound of the band in the building without the mics, and work from there. It rarely hurts to make an educated guess beforehand and have some mics already set up, but you really won't know what you need and what will work until you've heard the whole show in context -- and that includes the singers.
And this essay is long enough for the moment. Call this Part 1: in Part II I'll talk about sound effects for musical theater.