High Xorg cpu usage?

[bobthebob1234]

So I have 3 screens and 2 graphics cards but I don't think (k)ubuntu 11.04 is using them to their full advantage.

At the moment with 13 program windows open Xorg is using 79.7M and 15% of my CPU (AMD Phenon II X6). Also changing virtual desktops is a bit laggy and when I do get to one there are holes in some of the windows (where the previous window is showing though). Moving the window around gets rid of them usually

Also having more than about 4 pdfs open in okular almost freezes my computer, closing some sorts it out.

I don't think I am running out of ram (got 16gb...)

I have the NVIDIA 270.41.06 driver installed, and there is a more recent version on their website (there is a package for 12.04 in the repositories). Would it be a good idea to download and do a manual install of it? Or upgrading to 11.10?