I also have issues with it taking a long time to connect to servers, or not managing to connect at all.
When I try to connect it shows the load screen with an empty progress bar, which remains like that for 10 - 15 minutes until it quickly fills up and lets me in.
Other times it will do this and give me a connection timed out message, or I'll get timed out before it reaches the load bar screen.
Then every now and again it will connect me in just a minute or two.
Also I note that if I crash the game during the long connection wait and load back into the Multiplayer screen, the Server list will show that I am on the server and have been for a few minutes, even though I was stuck at the load screen during that time.
So far I've tried disabling any additional network adapters on my computer, and also turning off my anti-virus while I try to connect, but it doesn't seem to make any difference.