Novak Djokovic has a tough road to haul following the Olympics. He defends the Canada title - 1000 points. If he makes it to the final at the Olympics, he'll have no rest for Toronto, as he'll be flying with the medal back to Serbia, and then straight back to Toronto. The pressure will definitely be on him to win in Toronto. I can't see him skipping that tournament. Then Cincinnati is the following week - again, no rest. He defends 600 points there, and then a week's rest and the US Open where he defends the title. Then there is Davis Cup.
Federer will likely skip Toronto as he only defends 90 points there, and start his hard court season in Cincinnati, especially if he does better at the Olympics than Djokovic.