It can depend on a couple of other things too. If your network link to Google isn't good, things will be slower too.
Also, if you have networking layers like the Google Earth Community or Street View active, those will be fighting with the image data to get downloaded. I keep all of my network links and overlay placemarks turned off until I want to see them. Having the Terrain layer turned on is another big hit on performance.
I don't know how much that will help, though.