Pictured here is the Picotux 112, one of the smallest devices running on Linux. Yes, it is a server, a web-server to be exact. On the website, the younger sibling of Picotux 112 is spec'd as follows:

The picotux 100 is the world's smallest Linux computer, only slightly larger (35mm×19mm×19mm) than an RJ45 connector. Inside, there is an ARM7 CPU at 55 MHz running uClinux kernel 2.4.27 and Busybox 1.0. Two communication interfaces are provided, 10/100 Mbit half/full duplex Ethernet and a serial port with up to 230.400 baud. Five additional lines can be used for either general input/output or serial handshaking.

Whilst you can have Linux on a USB thumbdrive, you still need a computer to boot it up. This one only has the power supply as the biggest component - everything else is in that tiny device. Cool, eh? Imagine the possibilities.