I’m in a very similar position. When I found out about asexuality 7 years ago I initially embraced it as it explained how I was feeling, (or not feeling) but as time has gone on and I’ve reached 40 years old I now realise that I have let myself drift away from the few friends I do have because we have less and less in common, with me being eternally single (I’m aromantic too) and them all being married with kids.
I enjoy my alone time but the loneliness is creeping in more and more these days and it’s extremely difficult to reconcile the two.