My neighbor sat and watched me. Was I rude in confronting him?

DEAR MISS MANNERS: I moved into a house about three years ago. Late one evening, a male neighbor came over and cut my grass. Not expecting this, I gave him what little cash I had on hand, plus some refreshments. A few weeks later, I took a selection of holiday treats to his wife.
Afterward, the man began to come outside every time I worked in my yard. He’d sit and watch, walk to the end of the yard or driveway and even get into his car for a better view. He never greeted me or spoke when we made eye contact, which was weird and uncomfortable for me.
I tried to ignore it — until he approached one day when I was bent over with my back to the street. I turned and suddenly he was there without a word, so I yelped in surprise.
He left so quickly that I didn’t see him go. I was shaken and decided that it was time to have a discussion about his behavior. I changed clothes and went across the street to speak with him.
I asked what he’d come over for. He said that he wanted to tell me that I was working too hard, cutting hedges with a clipper instead of a trimmer. I said that working in my yard doesn’t bother me.
I asked why he watches me so much and also mentioned that he doesn’t speak. He replied by saying that he is nosy, wants to see what I’m up to and concluded with, “I guess I could wave, or mind my own business.”
I asked him to mind his own business and told him that he had been making me feel uncomfortable. Was I rude for this?
Miss Manners: My neighbor sat and watched me. Was I rude in confronting him?