I dunno it's only me, but I typically pose one actor, then I switch to another, then I pose that one. So when I'm posing actor "A" it makes no sense the thing will select actor "B" that is behind that one.
Words of wisdom right there folks! If a prop, character, or even a body part is not being worked on, it has no business being in the scene.
Here's a good example:
V4 is standing behind Terai Yuki. She has already been minimized, and yet the center point of her left arm is smack behind Yuki's head, making it hard to select it intuitively (not with some workaround drop down menus). So it (V4's left arm) too has no business being in the scene.
Don't blame the brush if you can't wield it!