What are the limits of my rights against trespers?

My wife and I live in a subdivision, we really like our home but we have one major problem, there is a lot of foot traffic that pes from one street to another right through our yard. I don't want to be at work and my wife be home alone with our baby and her be in the back yard and turn around to be startled by a group of men walking right past her. I have put up signs and called the police, who won't do anything, I have even put up trip lines (with some funny results) and of course I have confronted them. There are a lot of different people who do this, but none of them are anyone I would trust or want on my property or wandering around my house and things. Not to mention after talking to them and demanding they not walk through my yard, they now often scream profanity as they do it especially when I'm not home. Is it illegal for me to lay hands on these people.