A close associate of David Cameron apparently called Tory party activists "swivel-eyed loons". Swivel-eyed originally referred to squinting, but modern references are always related to people who are wacky or bonkers, particularly right-wingers. One of the citations in the OED for the entry 'barmy army' is "During the 1995 Tory leadership contest it was Collins who described John Redwood's campaign team as a ‘swivel-eyed barmy army’". That citation is from 2002 (Collins was a spokesman for John Major, and the quotation continued with ".. from Ward 8 at Broadmoor"). Another example, the phrase "“swivel eyed Daily Mail reading loon” appeared on this BBC forum last year.
Loon has several different senses. Actually, the sense meant in the phrase "swivel-eyed loons" is not mentioned on the loon page, but is closer to loony, which is short for lunatic. Loony is defined as "extremist, fanatical" and, ironically, became frequently used to refer to the "loony left" in the 1970s and 80s. One of the definitions of loon itself is "A boor, lout, clown; an untaught, ill-bred person", and is described by the OED as Scottish or northern English.
Loon is also an aquatic bird. The OED says that here loon developed from the word loom, a word from the Shetland dialect, and ultimately from Old Norse.
If you were around in the 1970s you may remember wearing loons ie flared cotton trousers. The noun loons developed from the 1960s verb 'to loon', meaning to lounge around or spend one's time pleasurably. The origin of this sense is unknown.