Actually if this happens, whoseever ball it is that rolled over should wait until our point is played out right? We are after all in the middle of a point here, right before a second serve. And if the server goes to get a ball that is not interfering at all - like the one sitting in the back, its at the servers own risk. I wouldn't give two - but then I don't know if I can have the guts to be like 'hellzz no' like I want to in a situation where, you know, its tough. Like if the opponent is a friend or friendly guy, or if you are winning clearly or losing clearly .. i mean often depending on the trend and just the person that I am, if they asked, I'd give it to them but in general I don't think you should.
If the ball is really interfering, maybe its valid. I personally never ask for a first nor have I had anyone ask one.
But like in your situation.. down 4-5 40-30, that's critical there. he has the guts to disrupt your rythem and ask you for a first? crazy, of course not.