Perplexed by my Virgo man, need help!

Ok so I met my Virgo about a year and a half ago. Things started off slow but after about 3 months we became exclusive. He's not the most romantic guy, but he went above and beyond for holidays, my birthday, etc. He would show his romantic side and tell me how much he enjoyed having me around. He has little pet peeves that would probably annoy most people, but I think they're cute. He's 4 yrs younger than me but so mature, reserved, organized, funny and smart - the qualities that have been lacking in my previous relationships. He's also the only guy that has ever noticed my freckles lol.

We were getting along great until about 7 months in. I used to talk to him everyday but then there was a time when I didn't talk to him for about 4 days. He didn't text or call like he used to and we started getting into minor arguments. I remember during one argument he mentioned that he loved me and I didn't even acknowledge it. Problem is I don't know how to acknowledge it - we both have a hard time communicating our feelings verbally. But I know/feel that we have a deep connection because of the way that he looks at me. He always looks at me directly in my eyes, like he's figuring me out or already has lol. He also reacts funny whenever I see him, his body language and face says all the things that I don't think he could ever tell me. It's hard to explain. My mind and heart tell me to leave him alone, but my gut tells me to stick with it because I think we could be great together.

Anyways, we broke up - supposedly - because I suggested taking space. That was about a year ago and we still text every once in a while. I feel like he's afraid of me - like he's scared to open up. Then sometimes I feel like he secretly loves me but just isn't ready. All I know is that I've always wanted him, I just wish that he would let things fall in place instead of fighting everything! urgh....should I give up and leave him alone......or keep trying to understand him?