I've only had this problem with fastballs, and it has been a problem since pinpoint was introduced. The straight down then up gesture likes to turn read red on the up portion of the gesture.
I would love to believe it is my controller, but as someone who owns PGA2K23, a game that relentlessly relies on this gesture, I'd be seeing negative results there as well if it were a controller issue.