TexasCHLforum.com

In most areas on land lots in Texas there is a 5` easement on each side of the property line for non-building zone and ingress, egress work area kinda like a no-mans land. Not all lots have this but many do it is for things like this so people can work and move around the property with as little a restrictions as possible. Generally utility's will run in that easement like power lines and fences between property's.
I am not saying yours is that way but many are but still workers should make every effort to let you know they need in the area before they just go on it.

Being a good neighbor is generally the best policy for everyone and it sounds like you are as long as the workers do their part and ask!

The worker challenging you to make them get off would have caused me some alarm also so calling LEO was the correct thing to do.