Cast solid actors who take pride in being good at the rough and tumble. Ask them outright at the audition how they'd feel about working under stressful conditions, and see if they flinch. I did this with an actor and he came back with the right kind of answer: "Don't worry about me. I was a Boy Scout."
Once cast with troupers, the best way for you to get that great performance on the day is to have had adequate, quality rehearsal time. This is not only for the obvious reason: that your actor will have a clear, well-prepared approach to their role, but also for the sense of security and camaraderie it can establish so the environment in front of the camera is more readily accessible to creativity. I'm not saying there should have been a lot of socializing beforehand; I think that tends to be extremely overrated, and even detrimental if it takes the edge off creative competitiveness. And, please... I don't mean talking and talking and talking about the part. If you don't know how to talk about acting in very specific, actable terms, it's better not to talk about it at all (and you should get my book). Just talking intellectually about a part will often do more harm than good; whether it's by making your performer mental about what it is they're expected to act, or by having them become suspicious about your qualifications to lead and if you're not going to be more of an adversary than an ally.
Have the actors rehearse and encourage them to try different things; even if you know that you'll eventually be having it done in a very regimented, pre-planned way. This encourages an overall freedom and confidence in a performer, and, who knows, you just might find out something you hadn't thought of. Imagine that.
Olivia Wilde & Me.
Do improvs on the scene and on scenes that aren't even in the movie. Take your talent on pertinent field trips and search for people or things that typify their parts, aware that the search is everything, not the finding. See if you can't present yourself to the cast as a combination of someone who has a very firm grasp for what they want, charged with high expectations, and an open collaborator, eager to welcome their deepest inspiration and wildest risks.
Let's say, however, you're there on the set and you feel a performance you're getting isn't what you'd hoped for; that the actor just isn't achieving what you think they might... the light is fading, you've got to make a company move and, ah hell, the cops are coming! What do you do? Don't panic... Well, all right, panic if you want, but for god's sake don't let your actors see that you're anything but assured that they're going to come through with flying colors. In fact, act as if they've already coming through with flying colors and you'd like to try some extra things for fun, just so you'll have options in editing. Making your talent feel they're letting everybody down is rarely going to get them to perform at their best.
There's an approach directors will sometimes take that I personally don't recommend, and that is to take the blame on themselves and say something like: "I know I'm not communicating this clearly and it's my fault this isn't working." Although that may or may not serve to make your actor feel less punished, it won't especially inspire them knowing that their leader is an incompetent ninny who's put the responsibility for rescuing this whole thing on their shoulders. I suggest you keep projecting a positive outlook and act as if this is exactly the way you'd chose for it to go in order to break through to the really amazing stuff. Make out that any awkwardness on your part is actually a deliberate technique for getting them to find it for themselves organically.
There are plenty of big-time directors who've been known to give bizarre directions, with perhaps only the intention of short-circuiting the performer's left-brain. You could be one of those. During the filming of The Thin Red Line Terrence Malik shouted out at Tim Blake Nelson: 'You're a squid that's washed up on the beach and you've used up all your ink." On the movie Go, Doug Liman told Scott Wolf: "You're a mountain lion pretending to be a bear."
This is not only for the obvious reason: that your actor will have a clear, well-prepared approach to their role, but also for the sense of security and camaraderie it can establish so the environment in front of the camera is more readily accessible to creativity. I'm not saying there should have been a lot of socializing beforehand; I think that tends to be extremely overrated, and even detrimental if it takes the edge off creative competitiveness. And, please... I don't mean talking and talking and talking about the part. If you don't know how to talk about acting in very specific, actable terms, it's better not to talk about it at all (and you should get my book). Just talking intellectually about a part will often do more harm than good; whether it's by making your performer mental about what it is they're expected to act, or by having them become suspicious about your qualifications to lead and if you're not going to be more of an adversary than an ally. 
