They were always a group in more ways than one. Before James got together with Lily, and left Remus and Sirius to be a couple rather than two of three, they shared everything.
Remus is sorry, now, that they never invited Peter to join them. True, none of them considered him their type, at the time, but it would have made things easier. They wouldn't have had to find places to go, since the four of them were the only ones in that dorm.
Still, what's done is done. But if Remus had known Peter was going to die so young, he would have done things differently.