What do you mean by slow/near freeze effects. Does the game actually freeze, Hulibert...?
Slowdon in a game like Oblivion is to be expected, if you ask me it doesn't really affect the game too much, other than in extreme cases.
Your saves could be the problem, if there really big then while the game is trying to access the large amount of data the game will suffer for a short while (especially when opening gates).