According to me (don't call me Jim,) the most annoying enemy in Oblivion was the cave / forest troll. It was big and incredibly fast, not to mention hard to kill unless you had a firespell at the ready. I'm pretty sure that it was meant to be scary, but it was mostly annoying.They do make a return in Skyrim, although seeing them now turns everything around - They're still fast, but not as in the annoying way in Oblivion; It is bigger; It will definitely be hard to kill, I'd say!
We've seen video footage of it's icy counterpart, and it is at least terrifying enough to make Hugh Grant cry - Not that he sets the bar very high.