How can your disk be full when df shows otherwise? "Inode" the answer!

Sorry, I just couldn't resist.  That's probably the only time I'll ever get to use that joke.  I had a situation recently where I had a system that kept giving errors that the root volume was full, yet df -h showed plenty of available space.  After doing a bit of digging, I found that there were hordes of tiny files chewing up inodes far faster than the space on the drive.  Essentially, this would be like filling a file cabinet full of folders with only a post-it note inside each folder.  The majority of the volume of the cabinet would be taken up with folders rather than "data."

I could detail how to diagnose and fix this issue, but someone has already written an excellent blog post on how to do just that.

No space left on device – running out of Inodes

Thanks to Ivan Kuznetsov for taking the time to post his tutorial!

Three Sysadmin Rules You Can’t (And Shouldn’t) Break

Saw this posted on the local LUG mailing list, and it's a must read!  I think I may have to print this out and put a laminated copy on the wall at work.

Three Sysadmin Rules You Can’t (And Shouldn’t) Break