Another problem with it being considered a rich man's sport is that the sports stars are looked at as being people born with silver spoons in their mouths. I don't think most of working class America likes the idea of cheering for rich guys wearing shorts and hitting a little fuzzy ball.
Plus, tennis players are looked at as being skinny pencil necked rich white guys, which is doubly bad for the typical insecure manly macho American middle class male. Sure, he has no problem expressing his admiration for 6 feet 7 inch tall guys who can kick his butt but I don't think it's as easy for them to openly tennis players. They see beefy guys tackling and hitting homers and dunking and they imagine themselves doing such heroic things, they are heroic vicariously, but I don't think most guys can relate to hitting a drop volley or a nice passing shot.
Another problem are boring, static camera angles, which for the most part have remained the same since 1778. Sure they do cool stuff with the camera every so often but, still, for 90% of the match it's the same shot of the entire court from the point of view of the peeps in the nose bleeds. Not very cool.
The scoring is also a problem, for TV scheduling, advertisers, and the casual fan, to which the scoring is incoherent. There's nothing intuitive about tennis scoring and it turns off fans.
Also, people can't cheer for their "home team" in tennis. And there are no jerseys and other identifiable logos that people like to wear around, promoting their affiliations and the game itself.
One of the reasons I think golf is more popular is that it's instantly more addictive than tennis. Sure, you can play like absolute crap the first time but eventually you're going to get that little ball into the cup even if it takes 15 strokes. So you get a sense of satisfaction every few minutes. Tennis is hella fun, too, but the first time you play tennis is mostly about picking up the ball and laughing at the ball going over the fence and giggling at completely whiffing the shot. It doesn't really get super fun until you start getting better. Not so with golf, which is fun no matter what your skill level is.