Plusnet Service

We are receiving an increase in the volume of customers reporting that they are not receiving emails since this morning. This could well be related to this:

http://usertools.plus.net/status/archive/1187607925.htm

If you were logged in to the portal, you could associate your username with this problem.

Problem has been passed to a network engineer for further investigation.

This problem is caused by the criticalpath trial. The problem is that the criticalpath boxes have been accepting all the mail without any problems, but then they have not been pushing it to our mail platform fast enough - resulting in high queues.

Criticalpath have stabalised the throughput but are now working to get the levels high enough to clear down the queues.

We have now rolled back the critical path configs on the firewalls. Critical path have been informed of the decision and are now working on delivering the queued mail that remains on their boxes.

Critical path are going to re-deliver all messages not marked as spam first (as 80% of our mail is spam, at least), thus getting customer mail through to them asap.

We have been made that some customers were continuing to experience delays in the arrival of email sent yesterday evening following confirmation that the problem had been fixed.

We are also aware that some customers are still awaiting email from yesterday day that was sent after the installation of the spam filtering appliance.

We are continuing to investigate and will provide further updates throughout the day.

We now believe that approximately 10% of legitimate email that was de-queued from the new spam appliance to our network was incorrectly recognised as spam and black-holed. Additionally the same percentage of new legitimate mails received between approximately 5pm and 11pm last night have also been deleted. These mails will not be recoverable and we recommend customers who have not received email that was sent yesterday to ask the sender to re-send where possible.

We are continuing detailed analysis of the events leading to the occurance of this problem.

Problem is being left open whilst we prepare the incident report detailing the events leading up to the problem.

For the time being further details can be found over on our discussion forums:-

http://community.plus.net/forum/index.php?topic=991.msg9225#msg9225

Further updates will follow tomorrow.

There are still around 200K messages that need to be dequeued from the critical path boxes. Most of these messages are likely to be spam.

During peak hours the mail platform is very busy so we expect most of these queued emails to be sent overnight with any that are remaining being delivered over the weekend.

We will deliver the incident report early next week.