I hang out with a lot of non-native speakers of Japanese. The conversation usually goes just fine, but every so often you run into words you can’t remember due to lack of use, and things like this happen.
Another thing that happens is that you start picking up the weird language habits of your friends, like the mistakes you repeatedly hear them make or phrases they fall back on a lot. Even though I know it sounds weird, my brain still picks them up! Thankfully the same thing happens with native speakers. My keigo has improved by leaps and bounds by hanging around a crowd that practices things like tea ceremony and kimono, but sometimes I catch myself sounding like an old lady.