Sev 3 - Inc 4613090 - Intermittent eCommerce payment capture failures

» View Event Details | Created Wed, 03 Aug 2022 11:59:00 +0000

Resolved

We have had no further reports of the issue where store colleagues were intermittently unable to finalise orders due to payment captures failing. We have identified all impacted transactions and have put measures in place to process these manually. The root cause is still being determined and we have raised a case with ACI to advise on this issue. Following the case notes, a Post Incident Review will be undertaken if required. No further updates.

Posted: Wed, 03 Aug 2022 15:17:00 +0000

Investigating

Between 10.30 am and 12.01 we received reports from some Stores advising that they were unable to finalise orders when capturing payments. Payments for selected orders were failing resulting in some orders being cancelled. This was not impacting the online customer journey when placing the order itself.

The issue seemed to resolve itself whilst investigations into the cause continue. Impacted stores should re-try capturing the payment for picked orders as most of these are now returning as successful.

Internal teams are compiling a list of any outstanding orders with the intention of manually capturing the payment on any unfinalised orders.

Raising issue with ACI and internal teams for root cause.

Next update 4.00pm or earlier in the event of a significant update.

Posted: Wed, 03 Aug 2022 11:59:00 +0000

Sev3 Incident - 4630051 - PPS Timeouts

» View Event Details | Created Mon, 15 Aug 2022 08:59:00 +0000

Resolved

PPS continue to investigate the issue with Microsoft Azure which caused degradation to the Offers service this morning. The workaround implemented by PPS has ensured our members have been able to load and select their offers. We now await the Root Cause Analysis report from PPS and will hold an internal Post Incident Review. No further updates.

Posted: Mon, 15 Aug 2022 15:49:00 +0000

Update

PPS have advised us there is internal an issue with Microsoft Azure that is still under investigation. To help mitigate this problem PPS have implemented a workaround by adding additional resources to help increase capacity for the push notifications. This means our members will be able to load and select their offers. We will continue to monitor performance throughout the day. Next update 17:00.

Posted: Mon, 15 Aug 2022 10:49:00 +0000

Investigating

We are aware of an issue with slow response times and timeouts when Members are trying to view and select offers. More than 50% of requests are timing out. This has been raised with PPS our third-party supplier and we are awaiting an update from them. Next update will be at 12:00, or sooner if the situation changes.

Posted: Mon, 15 Aug 2022 08:59:00 +0000

Sev 2 4632665 - Co-op Membership Sign - in, Offers and Reward unavailable.

» View Event Details | Created Tue, 16 Aug 2022 14:25:00 +0000

Resolved

After a period of monitoring we have no further reports of any issues. We're confident service has been restored and customers are able to sign-in to the Co-op Account (account.coop.co.uk) and Online Shop (shop.coop.co.uk). A Post Incident Review will be held to understand contributing factors and resolution. No further updates.

Posted: Wed, 17 Aug 2022 16:23:00 +0000

Update

We have continued to monitor stability for our services overnight. Customers are able to sign-n to the Co-op Account (account.coop.co.uk) and Online Shop (shop.coop.co.uk). The team are investigating the downstream impact following a configuration change that was rolled back to restore our services. Next update 17:30 or sooner if one is available.

Posted: Wed, 17 Aug 2022 11:14:00 +0000

Update

The issue where customers were unable to sign in to Co-op Account (account.coop.co.uk) and Online Shop (shop.coop.co.uk) has been resolved. This issue was caused by a configuration update within our services which has now been backed out and is pending further investigation. This incident has been placed into watch status. Next update 17/08 - 10am

Posted: Tue, 16 Aug 2022 16:16:00 +0000

Update

We are currently still experiencing an issue where customers are unable to sign in to Co-op Account (account.coop.co.uk/) and Online Shop (shop.coop.co.uk). Investigations into the cause are still ongoing. Next update 5pm

Posted: Tue, 16 Aug 2022 15:08:00 +0000

Investigating

We are currently experiencing an issue with the Membership service on line and in the application where customers are unable to sign in, undertake transactions, see, spend and earn reward. This will also be impacting customers signed into the the Online Shop service where customers may only be able to continue as a guest. Investigations into the cause are ongoing. Next update 4pm

Posted: Tue, 16 Aug 2022 14:25:00 +0000

Sev 4 - 4636925 - Amazon picking issue

» View Event Details | Created Fri, 19 Aug 2022 14:02:00 +0000

Resolved

Issue upgraded to P3 on 4641966. Continuing comms under new incident.

Posted: Mon, 22 Aug 2022 14:35:00 +0000

Update

The issue where some stores are unable to pick orders for Amazon customers has been investigated by a number of Co-op teams we believe this to be a DNS network issue with BT. Stores IT are deploying a workaround on the devices for the impacted stores and the root cause will be pursued with BT next week. Next update Monday 22nd August 10:00 or sooner if the situation warrants it

Posted: Fri, 19 Aug 2022 15:26:00 +0000

Investigating

We are aware of an issue impacting some Amazon stores, affecting colleagues ability to pick orders using handheld terminals. Amazon have disabled the delivery slots for the impacted stores to prevent customers placing orders. Teams have been engaged to help investigate the issue. Next update Monday 22nd August 10:00 or sooner if the situation warrants it.

Posted: Fri, 19 Aug 2022 14:02:00 +0000

Sev 2 - 4641966 - Amazon/Starship issue related to Utility Wifi

» View Event Details | Created Mon, 22 Aug 2022 14:36:00 +0000

Resolved

Following a period of successful monitoring over the past 24 hours there have been no further major network issues reported. This incident will now be resolved and a PIR scheduled. No further updates.

Posted: Fri, 26 Aug 2022 14:49:00 +0000

Update

BT have now successfully changed the DNS for all the remaining stores. The change and stores have been monitored for any issues and so far there has been no impact from this change, they will continue to monitor this impact. Next update 4pm tomorrow or sooner if the situation warrants it.

Posted: Thu, 25 Aug 2022 15:57:00 +0000

Update

5pm yesterday (24/08) this incident was upgraded to a P2 as it was identified that two other store services utilising Utility TPA (Deliveroo and In Store Radio) were also at risk. At 10:00 today BT Networks implemented a DNS change to 130 stores to rectify the instability of the Utility Wifi impacting Amazon and Starship stores. The change and stores have been monitored for any issues and there has been no impact from this change. Based on the success of this change, at 14:00 BT will progress changing the DNS of all the remaining stores. Next update 5pm or sooner if the situation warrants it.

Posted: Thu, 25 Aug 2022 12:59:00 +0000

Update

BT have identified the cause of the issues relating to the instability of the Utility Wifi impacting Amazon and Starship stores from picking orders. BT are raising an emergency DNS change planned for Thursday 25th August which will target all food stores this is pending approval. Next update tomorrow at 12pm.

Posted: Tue, 23 Aug 2022 14:49:00 +0000

Update

BT have identified DNS issues relating to the instability of the Utility Wifi in stores they are working to resolve. This means impacted Amazon and Starship stores continue to pick orders at a reduce capacity. Next update 4pm.

Posted: Tue, 23 Aug 2022 11:01:00 +0000

Update

Investigations continue into the issue impacting Amazon and Starship stores affecting colleagues ability to pick orders using handheld terminals. The workaround in place means stores can still continue to pick orders at reduced capacity. A case has been raised with BT to help get us to a resolution. Next update tomorrow at 12pm.

Posted: Mon, 22 Aug 2022 15:57:00 +0000

Investigating

We are aware of an issue impacting Amazon and Starship stores, this is affecting colleagues ability to pick orders using handheld terminals. Investigations have been ongoing since Wednesday 17th August by the networks Team who have made manual DNS changes to the device settings. This means stores can continue picking orders albeit at reduced capacity. We believe the issue is linked to the Utility Wifi in stores. Teams have been engaged to help further investigate the issue. Next update at 17:00.

Posted: Mon, 22 Aug 2022 14:36:00 +0000

Awareness - Kronos Maintenance 27th August

» View Event Details | Created Fri, 26 Aug 2022 15:13:00 +0000

Resolved

The change performed by 3rd party UKG on Saturday 27th August to the Kronos system was successfully completed.The Shifts service is fully available for colleagues. No further updates.

Posted: Tue, 30 Aug 2022 09:11:00 +0000

Scheduled

Between 12am-6am on Saturday 27th August, 3rd party UKG (Kronos) will be performing some scheduled maintenance. No full outage is expected, though there may be some periods of disruption to service during this period where colleagues will not be able to access Shifts to view/amend their rota.

Posted: Fri, 26 Aug 2022 15:13:00 +0000