Repairing a hacked WordPress installation

I have updated this post based on followup investigations.

This morning I found two independent reports of virus warnings being given for two different websites in the Ozblogistan network.

Given that I upgraded to WordPress 3.4.1 only 2 days ago, it would appear that a 0-day exploit is abroad in the wild.

As a tip for fellow WordPress victims administrators, here is the quickest way to detect and repair attacks based on modifying WordPress PHP files.

1. Login to your server with SSH

You will need a command line for what comes next.

2. Obtain a clean copy of WordPress

Go to the WordPress website. If you’re on the latest version of WordPress, you can use the magic Latest link to obtain a copy. Or you can look in the Release Archive for older releases.

Debian-based Linux distros store their website data in /var/www. For safety we will work on the clean copy from within our home directory — this will prevent the possibility of accidentally overwriting the operation copy of the code.

  [email protected]:/home/admin $ sudo wget http://wordpress.org/latest.zip
  [email protected]:/home/admin $ sudo unzip latest.zip

You will now have a directory wordpress in your home directory.

3. Use diff to look for abnormalities

diff is a mainstay of the Unix/Linux programmer’s toolkit and is commonly used to compare different versions of a single file. But diff can also be used to compare directories.

The key arguments we will use are:

r
Compare two directories recursively
b
Ignore changes in whitespace
q
“Quiet” output — just show us which files are different.

So now we type:

  [email protected]:/home/admin $ diff -rbq wordpress/ /var/www/wordpress/

We get back a great deal of stuff. What we’re looking for is files that differ between the copies.

Suppose for example that we get a report that wordpress/index.php and /var/www/wordpress/index.php are different. The next thing to do is to turn to diff in its standard file-vs-file mode:

  admin@example.com:/home/admin $ diff wordpress/index.php /var/www/wordpress/index.php

And we see an interesting difference pop up:

<?php eval(base64_decode('JGlwPSRfU0VSVkVSWyJSRU1PVEVfQUREUiJdOyRkcj0
kX1NFUlZFUlsiRE9DVU1FTlRfUk9PVCJdOyRc05qWTJMRFUxTUN3e2
ZXbG1LSFlwWlQxM2FXNWtiM2RiZGlzaWJDSmRPM2M5Wmp0elBWdGRP
M0k5VTNSeWFXNW5PM285S0NobEtUOGlRMjlrWlNJNklpSXBPMlp2Y2
lnN05UYzFMVFVyTlQ1cE8ya3JQVEVwZTJvOWFUdHBaaWhsS1hNOWN5
dHlXMlp5S3lnb1pTay9Ja052WkdVaU9qRXlLVjBvS0hkYmFsMHZLRF
VyWlNnaWFpVXlJaWtwS1NrN2ZTQnBaaWhtS1dVb2N5azdmVHd2YzJO
VGdzTlRBMUxEWTVOaXd6TkDQp9'));?>

This is a scrambled version of an actual piece of malware I found.

The eval(base64_decode()) pattern is a dead-certain red flag for malware. There is no earthly reason for honest sourcecode to be stored in this fashion, it is always bad news.

Update: That’s not a complete account. In particular, this attack also inserted code into every theme template, which the above approach doesn’t reveal.

You also need to use grep to dig out instances of eval and base64_decode. Search for both independently, because some clever malware inserts random comments to obscure the combination.

4. Replace infected files with clean copies.

So you’ve found out that /var/www/wordpress/index.php is dodgy. Now what?

The simple answer is — replace it:

  [email protected]:/home/admin $ sudo cp wordpress/index.php /var/www/wordpress/index.php

Do this for each file you found in step 3.

Update: You may also find that files have been added. It looks now like I had two different attacks, or perhaps one attack with two mechanisms of operation. The diff -rbq revealed a series of files with random nonsense names, such as itrqwx.php, scattered throughout the file system. I deleted these as well after looking in them for malware.

5. Final cleanup

Bad code sometimes lingers in caches, so clear those out. I use wp-supercache and some nginx magic to cache .html.gz files on disk; those need to be flushed to stop serving stale, malware-infected pages.

  [email protected]:/home/admin $ sudo rm -rf /var/www/wordpress/wp-content/cache/supercache/*

I also use PHP opcode caching, so I will need to restart PHP as well:

  [email protected]:/home/admin $ sudo service php5-fpm restart

And that’s it. I have now scrubbed the installation of an annoying virus infestation.

Update: Do it in the opposite order. No point flushing the on-disk cache until you’ve flushed the running software, as the cache will immediately begin to fill up with new potentially-infected pages.

6. Prevention

It’s possible to prevent such attacks by locking down file permissions to deny PHP the ability to modify files. However doing so breaks a lot of the update capabilities of WordPress. My own scheme is that I allow PHP the ability to modify its own files, but no others. I have separate users for nginx and PHP and they are unable to modify each other’s files; and of course neither has sudoer or setuid capability.

It’s a calculated tradeoff. I’m relying on the WordPress project team to prevent abuse of the file-modification permissions from within their software. Given that Ozblogistan has twice so far been auto-hacked, their record is not that great. On the other hand I already have to do a lot of dashing hither and yon to keep my bloggers happy; locking down the files would only increase the number of things I have to do by hand. It’s an unhappy compromise.

Update: I changed my mind because the malware would aggressively metastasise whenever I deleted one part in isolation. Delete file X and file Y is quickly created to replace it. So for now I have locked all PHP files down to the lowest level of permissions that work: read-only for the owner, the group and world (444). That’s stopped the malware from reinstalling itself when I remove only part of it.

This entry was posted in Ozblogistan, Technical Notes. Bookmark the permalink.

12 Responses to Repairing a hacked WordPress installation

  1. Pingback: Skepticlawyer » Hacked again

  2. movius says:

    Just letting you know, I’m still getting the warnings from skepticlawyer

  3. Terry Dwyer says:

    My virus scanner (Sophos V9.5) was blocking me from Cat with the message
    Temporary Internet Files\Content.IE5\RRFIXQI2\catallaxyfiles_com[1].htm” belongs to virus/spyware ‘Mal/Iframe-W

    So i installed the free AVG toolbar for IE and it is reporting that
    “Catallaxy.com is infected with Blackhole Exploit Kit(Type 2170)

  4. When was this, Terry?

  5. Terry Dwyer says:

    It was doing it all Day yesterday (05/07/12)

    It is still doing it now at 11:56am on 06/07/12

  6. I’ve gone over the site again and removed some suspicious-looking files.

    That said, AVG is a holdout. Every other web scanning service (Google, Norton etc) is calling the site clean. I’ve submitted a form for the AVG service to rescan the site.

  7. Terry Dwyer says:

    Thanks Jacques

    I don’t know how long AVG took to clear it, but I just checked now 09/07/12 1:00pm AUS Eastern Standard time and it is all clear

  8. John says:

    I have a client whose web site got the blackhole 2170 on 8 .js files, some were word press and some were Frontpage Slideshow. Any idea how this exploit gets onto the server? They are running an older version of word press ver 3.1.

    Thanks,
    John

  9. John;

    I’m not sure exactly; however my guess would be that a vulnerability in WordPress is exploited to modify files on the server. In your case it skipped modifying PHP and modified any javascript file it could find.

  10. Tel says:

    Egats, now people all over probably have copies of my email address and will be putting false postings with my name all over the Internet, just to make me look bad. This very post might be fake for all I know.

    If you are going to use “diff” then might as well go the whole hog and use “git”. Bit of a counter-intuitive interface, but very powerful. Worth learning. Something along the lines of:

    cd /var/www
    git init
    git add wordpress
    git commit

    Then you can put a comment about this being an absolutely fresh installation, and put a version number, where you got it from, blah blah. After that’s all digested, leave a tag behind for future reference:

    git tag wp-x.y.z-foobar-woobar

    Later on you can see if anything changed:

    git status
    git diff

    If you do see changes and you are comfortable with the cause of those changes, go back around the earlier cycle:

    git add wordpress
    git commit
    git tag wp-x.y.z-changed-blah

    Now you have accepted the previous changes, taken a stride forward, and future diffs work from the most recent baseline you committed. There’s a lot more cunning stuff you can do, but plenty of documentation is already out there (don’t worry if it makes no sense, that’s normal, but surprisingly over time you just get used to what it does).

  11. Actually, I do use git these days. Mostly because when (not if: when) the WordPress guys fuck up or a plugin upgrade goes bad, it lets me back out the change almost immediately and with great ease.

    But it requires some discipline — mainly you need to consistently commit after updating plugins and adding new stuff.

    But your advice is sound.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>