Whether you are recording sound effects for theater or a video game, there are a handful of basic techniques and tricks that can help you get a better sound from your voice actors.
1) Get an actor. A voice actor is preferable. Random friends and people off the street may have an interesting sound that catches your ear (I really want to record the guy that sold me my previous car -- he has this wonderful combination of Russian accent and street hustler smarm that makes you want to ask if he can smuggle a nuke for me out of the former Soviet Union.) But they don't have training; many of them will choke up on mic, few can do a cold reading off a script, and fewer still can repeat or adjust their vocal performance.
An actor has the chops to pick up the script, fall into character, and read, and do it again and again on cue. And a trained voice actor? They can do incredible things. I worked with a guy once recording a radio announcer introducing a radio soap opera. I asked him for "Can you do the same thing, but with a little more Latin Lover and just a touch of condescension, but also speed it up a little?" And he did!
An actor will also show up on time, prepared, and will stay for the scheduled length of the session. Friends have a tendency to show up late, and get quickly distracted. Amateur actors are just fine, and they all over; in the schools, in community theater programs. It isn't hard to find them and if you pick up one through your social networking you might not even have to pay them.
2) Get a director. Just as an actor is a professional at saying the lines, a director is a professional at knowing how best to say the lines. A director will find the best drama in the lines. A director will inspire the actor, pushing them, working with them, responding to them. As you are huddled back by your mixer with your headphones on, the director becomes your point man, holding the talent's hand and keeping them focused.
If you are creating material for a specific play, having the director of that play in the room is practically mandatory. If the actor is in the show and pre-recording a line for some technical reason, the director needs to be there the same way she would be for any rehearsal; to make sure his character and performance stays within her intents as a director. If the actor is not in the show, a key vocal performance still becomes a character on stage. The director, again, choses to direct this unseen and pre-recorded character in the same way she would direct any live actor.
3) Print the lines. Double-spaced, generous margins, no smaller than 10 pica. An experienced actor will mark up their script with the necessary breaths, with accents they want, with the pronunciations required by you and the director. Use multiple sheets of paper as necessary so that each separate "shot" you mean to record will be on a fresh sheet. Do this even if it ends up you only have six words on a single sheet of paper.
4) Make the talent comfortable. Give them time to warm up, make sure they have water, enough light, etc. Encourage them. Give them feedback; react to the performance, talk back to them. When the lines come in context (if they are alternating parts of a conversation) have someone read the lines they are responding to.
5) Physicality. If the voice over is of a person sitting at a desk, then let the talent sit. Otherwise, encourage them to stand. It shows up in the voice! If you are trying to get a sound of someone in action, have them move. Have them act out, to within the limits of good vocal production and the needs of the microphone.
There is a great story from the conductor of the orchestral and choral material which was made available to designers of the Lord of the Rings series of games. He couldn't get the dwarf song to sound right until he thought of asking the chorus to rock back and forth from one foot to another. As he put it, it transformed the sound from an okay choral performance to the doleful working song of the dwarves laboring away their endless hours in their mines deep under the earth.
I once recorded a brief snippet of sailors pulling up the sails on a clipper and I had the actors join me in miming the pulling of a rope as we grunted out "heave-ho!" in unison.
This extends even to the choice of microphones. If you want a man on a telephone, then give them a mic they can hold and move in close on. If you want the sound of a person shouting across a chasm, stick the mic six feet away and MAKE them shout to you. This is one reason I love my large-diaphragm condenser, and my fishpole boom; they let me move the microphone out of the actor's eye line and force them to speak to ME, not to the mic. It makes a world of difference in the sound.
But I recorded all the ship's announcements for "Mr. Roberts" on a vintage "lollipop" mic I found on eBay and re-wired. To get sound out of it you had to grab the mic in one hand, lean over it, and speak LOUD. It was the perfect sound; even better, all that handling noise came through and sold the idea of a ship's announcement system being used by the crewmen even better.
6) Environment. You don't need an acoustically perfect space, but you do want to control it. Most of the utility rooms and empty rehearsal halls they will give you will have too many reflective surfaces. All that slap-back will show up in the final recording; no matter how much you try to hide it, the sound of a voice shouting across the empty desert will still sound like a voice shouting across a room.
I've made more than a few recordings in theater lobbies, as bad as most of them are. My secret weapon is costumes storage rooms; all those racks of clothing are almost as good as foam acoustic panels. A large cluttered space, like a scene shop, will do the same; all the old props and tool cabinets and bench tools break up those reflections and hide the nature of the room.
When the stars are right you can use the room's acoustics to your advantage. The best use I have made of this is simulation of space; I've recorded actors on the very stage we are using for the production. Played back through the house speakers, the re-creation of a phantom actor is uncanny. The human mind is very, very good at picking up clues about the shape and size and wall treatment of a room from the subtle reflections of sound. When you have that actual room, you can use this. Consider, if you want to record an actor in a car, sticking them in a car to make the recording!
I've recorded material where I've made use of deep space. For "Rosencranz and Gildenstern" I scattered my voice actors across the recording space and had them shouting back and forth. The recording then had the natural differences in level and spacial cues of the physical placement of the actors.
You notice I haven't said anything about gear. Personally, I currently use a couple of condenser mics, small and large diaphragm, and some speciality mics. I run them, these days, through a firewire interface that has phantom power and I record directly to hard disk on my Powerbook...usually using Audacity. When I have a chance I run several different mics at one time; in case there is a problem with one, or in hopes one of the alternates will have a sound I like better.
I've recorded plenty of stuff off my Sony Minidisc, though. With the noise floor of live theater, it is good enough, and it is much more portable.
In my pack are also a couple of tripod boom stands, shock mounts for the condensers, and a fishpole boom. I don't usually use pop gags...I probably should invest in one for my kit though. Lately I've become very enamored of the sound quality and flexibility of recording from a fishpole instead of tripod or hand-held. It gets you out of the actor's eyeline and gives them room to move around, and the high position controls breath pops and script noises.
I don't start and stop recording. I start recording the moment the talent gets the script, and I keep running until they leave, only stopping long enough to save the file.
Last things in the kit? Headphones, and if possible, powered monitor. Although no-one, including actors, likes how they sound on a mic, voice talent often asks anyhow if they can hear "how it came out." Directors, when you have one, tend to demand it. So having a playback system on you is a nice gift for them and will let them finish off the voice session with a feeling of a job well accomplished.
And that's when your job continues; to take the raw takes, identify the good takes, listen for problems, tame the wild levels, duck the errant noises, clip the extra-long spaces, and all that other editing that makes a good vocal session into a professional-sounding performance.