I frequently play against people that can serve aggressively on both sides. I am able to pick up clues somewhat over setup, ball toss, racket movement to help me out a bit, also if i notice that a person only serves in one spot, i cheat over.
I have trouble with people who have a good serve always being in position to get the serve, on some of the far wide or fast narrow shots I dont get to them. I consider myself fast as well.
I am getting decent at hitting serves back, once i am in positions, but once in a while, one will fly by me out of my reach. I try to start on the baseline near the corner of the singles when playing competitive serves and stay low. I notice that just turning my body sideways and doing a follow through with little to no backswing works fairly well as an aggressive return. For really fast serves, sometimes i just use a volley motion to get em back.