Yesterday's security incident

Any problems or suggestions regarding the forums or scripts, post them here.
Post Reply
User avatar
Miguel
Posts: 262
Joined: 02 Feb 2006 09:24 pm
Gender: Male
Location: Inside your computer
Contact:

Yesterday's security incident

Post by Miguel » 29 Feb 2012 01:45 am

Hi everyone, I'd like to explain the recent problems experienced by the site, resulting in Google labelling us as potentially malicious.

Firstly, the good news. I've manually been through the entire site to ensure that any malicious code has been removed, so the site now has a clean bill of health. I've re-submitted the site for review by Google, so the warnings for Chrome and Firefox users should disappear in the next day or so.

We're very sorry for any inconvenience this has caused, and happy to answer questions regarding the incident or any concerns you may have over your personal computer safety.

For those that are interested in what happened, I'll set out what I understand of the incident:

Earlier tonight, Jazzy and I received notifications via MSN from other forum users that Google was flagging the site as malicious (a big red scary warning screen). A very quick investigation of this confirmed that this was definitely the case, and that the site was visually functional but also trying to load malware onto visiting computers. We took the site down immediately on discovering this, and started the incident response and site restoration process.

Investigation showed that a large number of source code files across the site were 'infected' with short pieces of either php code or javascript code which were attempting to load the additional software from malicious third party websites. Whilst I haven't investigated the malware in depth, my quick, informal and most definitely not professional opinion is that anyone running a fairly modern browser and operating system with patches installed and up to date should not have gained any unwanted additional software from this.

If anyone on the forum is concerned that they may have malicious software on their system following this incident, contact me via private message or email nc@benmitchell.co.uk and I will assist.

More good news; on restoring access to the site using known good software, I checked the main NC database for signs of intrusion. There is no evidence to suggest that the database holding usernames and passwords has been compromised (please note we DO NOT hold passwords in any recoverable form anyway). The database runs on a separate server, with different login credentials to the main site or hosting account.

The malicious code present in the website source files appears to be automatically placed rather than tailored with human intervention, in which case it is unlikely that the database login credentials were taken from the website storage. In any case, access to the database server has to be made from the web server, which has been re-secured. Database credential changing for NC is non-trivial, but will be undertaken in the next few days as a further precaution.

Investigating a local copy of the affected files has not presented a single definite vector for the attack. My opinion, given the content and style of modifications made to the site, and the Google reports of the domains we were pulling malicious content from, is that the FTP credentials were compromised through an indiscriminate brute-force attack (we can't change the server name or user name, which are both linked to publicly identifiable information). If this was the attack vector used, then this could have been partially my fault - during a problematic system upgrade with the web hosts a few weeks ago, I changed the password to grant the host company access, and didn't change it after they were finished. Therefore, any compromise of the web hosts' system, or any copies of the plain-text emails with the password, could have resulted in the compromise of our account credentials.

Obviously, I have performed several actions to lower the likelihood of a similar event happening in future, including implementing a more sensible password policy, site account lockdown, and removal of legacy and maintenance applications. The site restoration has taken a bit longer than anticipated, as the latest complete offline (and thus safe) backup was not up to date, so some degree of manual cleaning and stitching together of files has been necessary to get back up and running.

However, we're back up and running within about 6 hours of first notification. We're very sorry for any inconvenience caused by the downtime or perilous warnings of peril made by Google, and if anyone does have any concerns about the security of their details or their computer following this incident, I am happy to provide more advice.

However, I have to be getting up in ... 4 and a half hours. So I'll tie up any loose ends tomorrow night (UK time), now that the site is safe, secure, up and running, and I've done the paperwork (reporting to you guys, after all, many of you are shareholders...). Goodnight!
Image
Many thanks to EofS for the avatar and signature images.

User avatar
AngharadTy
Zombie Queen
Posts: 5251
Joined: 08 Jan 2006 05:20 am
Gender: Female
Human Avatar: 89833
Location: Tyland
Contact:

Re: Yesterday's security incident

Post by AngharadTy » 29 Feb 2012 03:12 am

Thank you, Ben and Jazzy! I really appreciate how frustrating it is to deal with this as I had to deal with it myself before. So thank you ever so much for putting that effort in, and much faster than I did when I had that problem elsewhere. ;)
Image Image

User avatar
thelonetiel
Posts: 1065
Joined: 07 Jan 2006 08:56 pm
Gender: Female
Human Avatar: 15268
Location: Nuevo Mexico, Estados Unidos

Re: Yesterday's security incident

Post by thelonetiel » 29 Feb 2012 03:20 am

After reading this, you and Jazzy are such great webmasters. I feel terribly lucky to have such dedicated people at the background of my favorite haunt.

I was more than happy to be patient while things were sorted out, but I'm very happy to see the site back online. I'm also very appreciative of the great explanation of what happened. I'm glad things seem to be set right and without major issue.

Virtual toasts all around!

User avatar
Jessi
Posts: 3412
Joined: 09 Mar 2006 06:29 pm
Gender: Female
Human Avatar: 155904
Location: Seattle, Washington
Contact:

Re: Yesterday's security incident

Post by Jessi » 29 Feb 2012 04:32 am

If we had a rep system, I would +1 you and Jazzy to death, Ben. Thank you SO much for being so thorough with this and getting rid of it! I admit, when I came to NC earlier and saw the message saying it was down for security issues, I was really worried (thinking back to a few years ago when what's-his-name basically tried to take over NC!), but I'm glad it's all settled down <3 You guys are the best!

User avatar
Fjorab_Teke
Posts: 1702
Joined: 28 Jan 2006 10:38 am
Gender: Female
Human Avatar: 271433
Location: Tennessee or Georgia, take your pick
Contact:

Re: Yesterday's security incident

Post by Fjorab_Teke » 29 Feb 2012 04:49 am

Eek, between this and Neo having weird issues, it has me a little spooked, especially as i came and found the big huge warning redirect and told it this site isn't malicious.

Crazy.

But hurrah for you all working so hard to restore our little home on the web. You're awesome! Hopefully no real damage was done, just obviously a lot of hassle.

User avatar
Kantark
Posts: 1926
Joined: 18 Jan 2006 08:59 pm
Gender: Male
Location: UK

Re: Yesterday's security incident

Post by Kantark » 29 Feb 2012 09:50 am

Tiel wrote:Virtual toasts all around!
Hear, hear!


I didn't expect the site to be up and running this morning but thought I'd try it anyway, fully expecting the scary Google page of Death and... NC's back!

Awesome work, totally above and beyond the call of duty.

And you beat the Neoboards, they're still down :-)
Image
Neopets: sparkygoesforth, decommissioned, nightfall, LiveJournal:kantark, Last.FM:Kantark

User avatar
Miguel
Posts: 262
Joined: 02 Feb 2006 09:24 pm
Gender: Male
Location: Inside your computer
Contact:

Re: Yesterday's security incident

Post by Miguel » 29 Feb 2012 11:46 am

Whoops! I've just fixed the front page now, I forgot I'd left that ready to roll last night but then not taken the security notice down! Fixed, sorry!
Image
Many thanks to EofS for the avatar and signature images.

User avatar
Miguel
Posts: 262
Joined: 02 Feb 2006 09:24 pm
Gender: Male
Location: Inside your computer
Contact:

Re: Yesterday's security incident

Post by Miguel » 29 Feb 2012 08:34 pm

And in case anyone's concerned, Google have now given us a clean bill of health again.

Their assessment is at:

http://safebrowsing.clients.google.com/ ... /index.php
Image
Many thanks to EofS for the avatar and signature images.

User avatar
Color Wheel
Posts: 385
Joined: 06 Feb 2012 02:53 am
Gender: Female
Location: Neovia
Contact:

Re: Yesterday's security incident

Post by Color Wheel » 29 Feb 2012 11:07 pm

Thanks for getting everything sorted out so quickly! Glad to know everything's back to normal.

User avatar
Madge
Posts: 1542
Joined: 19 Jan 2006 05:05 am
Gender: Female
Location: Perth, Western Australia
Contact:

Re: Yesterday's security incident

Post by Madge » 29 Feb 2012 11:23 pm

<3 thanks guys!! We love you.

User avatar
Runic
Posts: 182
Joined: 23 Apr 2011 10:37 pm
Gender: Female

Re: Yesterday's security incident

Post by Runic » 01 Mar 2012 03:39 am

You guys ROCK. x3
Granted, when I visited the site unaware of the issue, I got hit with the Trojan, but my Kapersky protected me and prevented it from getting in.

I was so worried that someone DID hack NC. And then I read that Neo was having issues in their news features, and I freaked even more.
I thought Subeta would be affected next. xD;;

I'm relieved that everything was okay! Thanks so much Miguel and Jazzy. x3

User avatar
Jamie
Posts: 614
Joined: 08 Jan 2006 12:14 am
Gender: Male
Location: Australia
Contact:

Re: Yesterday's security incident

Post by Jamie » 01 Mar 2012 11:38 am

Echoing everyone else, well done! You guys handled it all very well and timely!

User avatar
Darigan
Posts: 334
Joined: 06 Feb 2006 03:34 pm

Re: Yesterday's security incident

Post by Darigan » 02 Mar 2012 12:17 am

Thanks a lot!
Just wish Neopets was that transparent regarding its issues.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests