I figured it out. Everyone needs to be friends on their respective platform of choice. Some will work but some players have restrictions.
I had the same issue until I figured out the two guys getting locked out were not friends. They sent requests accepted and they were good to go.
The buddy that was the culprit plays on pc from Xbox app or whatever.
Also, it appears cross platform friend requests do not matter.