Sev2 Incident - 4513218 - Stores unable to pick Ecomm orders

» View Event Details | Created Sat, 04 Jun 2022 10:19:00 +0000

Resolved

It has been confirmed that store colleagues are now able to access the picking app again. The maintenance page has been removed from shop.coop.co.uk and customers are able to place orders for delivery and collection as normal. No further updates

Posted: Sat, 04 Jun 2022 11:32:00 +0000

Investigating

Due to an issue with the Salford Fortinet firewall, stores are unable to pick and prepare orders placed on shop.coop.co.uk. This issue is currently being investigated by Data Centre Services and the MIM team. To avoid customers placing orders that cannot be fulfilled, a maintenance page has been deployed to shop.coop.co.uk. Next update: 14:00 or sooner if one is available.

Posted: Sat, 04 Jun 2022 10:19:00 +0000

Awareness - CR 565267 - Naveo Release impacting Food Ecommerce

» View Event Details | Created Tue, 07 Jun 2022 09:05:00 +0000

Resolved

The planned release for Co-op Online Shop was completed successfully at 05:18 within the release window. Initial tests are showing all services as operational. The teams will continue to monitor the service throughout the day.

Posted: Wed, 08 Jun 2022 08:20:00 +0000

Scheduled

The Food Ecommerce Service will be unavailable on Wednesday 8th June between 5am - 6am. This is due to a planned release to be undertaken by our 3rd party suppliers Naveo. Customers will not be able access or utilise shop.coop.co.uk whilst the picker app will also be unavailable to store colleagues during this time.

Posted: Tue, 07 Jun 2022 09:05:00 +0000

Awareness - CR 565346 - Failover GP and CRM from Rochdale to Salford (Funeralcare)

» View Event Details | Created Sat, 11 Jun 2022 16:00:00 +0000

Resolved

The Disaster Recovery test on the CRM (colleague system) successfully completed on Saturday. There was no impact to the Funeralcare website or data transfer to the down stream systems. The teams will monitor over the next few days to ensure service performance is optimal.

Posted: Mon, 13 Jun 2022 08:53:00 +0000

Scheduled

On Saturday 11th June between 4pm and 10pm a planned Disaster Recovery test will be taking place on the CRM (colleague system). During this test the Funeralcare website should remain available, however the information from Plan purchases, Direct Cremation enquiries or Urgent enquiries may be delayed getting to the CRM system. All relevant checks to ensure services are fully operational will be carried out post change.

Posted: Sat, 11 Jun 2022 16:00:00 +0000

Sev2 Incident 4534617 - Funeralcare Address Search Failing

» View Event Details | Created Thu, 16 Jun 2022 07:04:00 +0000

Resolved

Address look up service on the Funeralcare direct cremation and pre-paid plans pages have been stable since 08:51am. Customers can continue to use the address look up feature to select their address when entering their postcode. The team are investigating the root cause of the issue and a post incident review will be held. No further updates.

Posted: Thu, 16 Jun 2022 10:57:00 +0000

Update

The issue where address search was failing to return results for customer postcodes appears to be back up. Customers can now purchase a direct cremation or pre-paid plan on the Funeralcare website. A change has been made with Experian and service has been restored. Teams will monitor stability this morning. Next update: 12pm

Posted: Thu, 16 Jun 2022 08:15:00 +0000

Investigating

We are currently investigating an issue where the address search is failing to return results for customer postcodes. Customers will not be able to purchase a direct cremation or pre-paid plan on the Funeralcare website. Next update 9am.

Posted: Thu, 16 Jun 2022 07:04:00 +0000

Sev 2 Incident 4540365 - Membership Wallet Unavailable

» View Event Details | Created Mon, 20 Jun 2022 07:53:00 +0000

Resolved

The Wallet service has been monitored overnight with no further issues observed since yesterday at 15:32. The CBS have also not reported any further issues and have been able to continue to adjust wallet balances and replace cards through salesforce. We are still awaiting updates from our 3rd party ACI a Post Incident Review will be held to understand contributing factors and resolution. No further updates.

Posted: Tue, 21 Jun 2022 08:58:00 +0000

Update

Connection to ACI has been steady since services were restarted, and wallet is still working as expected. We are still awaiting further details from ACI so we can establish what caused this incident and what can be done to resolve it. Next update 10:00 unless the situation changes.

Posted: Mon, 20 Jun 2022 15:53:00 +0000

Update

Some services have been restarted to restore a connection to ACI. Since then requests have come back to a more normal volume. We are awaiting ACI to investigate the issue and update us. Members should be able to view balances again and CBS should be able to make wallet adjustments and order replacement cards. Next update 17:00 or sooner if the situation changes.

Posted: Mon, 20 Jun 2022 15:04:00 +0000

Investigating

We have observed that wallet connections are intermittently unavailable again. This has been investigated and it appears that ACI servers are intermittently not presenting any certificates. This has been raised with our 3rd party supplier ACI to investigate. Next update 16:00.

Posted: Mon, 20 Jun 2022 13:58:00 +0000

Resolved

Following a restart of a couple of APIs we have been able to re-establish a connection with ACI and certificates are running again. Customers are able to see their reward balance and CBS have confirmed that they are able to replace cards and update balances. A PIR will be scheduled. No further updates.

Posted: Mon, 20 Jun 2022 08:46:00 +0000

Investigating

We are currently investigating an issue with the membership wallet it appears our 3rd party support ACI are not returning a certificate. This means customers will be unable to register, replace cards, view balance and there may be a delay signing into their membership account. The Co-op Business Services (CBS) desk will also not be able to replace members cards or adjust wallet balances in salesforce. The team are investigating. Next update 10am.

Posted: Mon, 20 Jun 2022 07:53:00 +0000

PPS Outage Awareness - CR565268 - Offers System Features Update

» View Event Details | Created Mon, 20 Jun 2022 10:08:00 +0000

Resolved

The PPS planned maintenance completed successfully last night with minimal impact to service. No further updates.

Posted: Tue, 21 Jun 2022 09:55:00 +0000

Scheduled

PPS - Our third party suppliers, are undertaking planned maintenance on their systems on Monday 20th June 11pm - 21st June 10am. There is an expected OUTAGE OF SERVICES BETWEEN 11pm and 1am (2 hours). Offers on website and the app will be unavailable for this period. The service is expected to function normally for the remainder of the change window.

Posted: Mon, 20 Jun 2022 10:08:00 +0000

Sev 3 - Inc 4546881 - Festival store slots are unavailable

» View Event Details | Created Thu, 23 Jun 2022 09:33:00 +0000

Resolved

The issue of Festival Store slots being unavailable has now been resolved. The fix was deployed and service was restored for these customers at 11.55am. No further updates.

Posted: Thu, 23 Jun 2022 11:03:00 +0000

Investigating

Customers are currently unable to select slots at Festival Stores making the Online Shop service unavailable for them. All other stores remain unaffected. The root cause has been identified and a fix is being tested for imminent deployment. Next update 11.30

Posted: Thu, 23 Jun 2022 09:33:00 +0000

Sev 3 Incident 4552236 - Offers not refreshed for some members

» View Event Details | Created Mon, 27 Jun 2022 07:38:00 +0000

Resolved

After PPS Re-imported a file with offers data last night we have seen that since ~7pm the number of members unable to see any offers has dropped back down to zero. This has been monitored this morning, and push notifications have started to go out without causing an impact to response times. A PIR will be scheduled. No further updates.

Posted: Tue, 28 Jun 2022 09:21:00 +0000

Update

We are still investigating the cause of this issue alongside PPS. A decision has been made to schedule push notification and email campaigns from 10am tomorrow, and PPS have been asked to scale up capacity in anticipation for the increased traffic this will cause. Next update tomorrow at 12:00 unless the situation changes.

Posted: Mon, 27 Jun 2022 16:05:00 +0000

Update

Further investigations have identified that PPS are sending relevant data for most cards but not for all cards in the generic offers campaign. We are awaiting feedback from PPS to confirm where the root cause is believed to lie and what can be done to fix the issue. Next update 17:00 or sooner if the situation warrants.

Posted: Mon, 27 Jun 2022 13:04:00 +0000

Update

The offers issue is still being investigated by PPS who have now been provided with example member details to support the investigation. The Personalisation team have confirmed that there are no issues on their end which would have stopped members receiving their offers today. Next update 14:00 or sooner if the situation warrants.

Posted: Mon, 27 Jun 2022 10:21:00 +0000

Update

The issue with offers is still being investigated by our third-party supplier, PPS. The push notification and email campaigns have both been paused to try and reduce traffic into offers. Next update 11:00 or sooner if the situation warrants.

Posted: Mon, 27 Jun 2022 08:57:00 +0000

Investigating

We are investigating an issue where approximately 3% of members who have attempted are unable to view any offers this morning. The issue has been raised with PPS, our 3rd-party supplier. Next update 10:00 or sooner the situation changes.

Posted: Mon, 27 Jun 2022 07:38:00 +0000

Awareness - Planned Change: CP-306 - Bulk migration/cloning of Grafana dashboards

» View Event Details | Created Tue, 28 Jun 2022 10:00:00 +0000

Scheduled

The Digital Engagement & Loyalty team have a planned change to migrate the monitoring logs on Tuesday 28th June 10am - Friday 1st July 5pm. We are expected to see alerts trigger across our monitoring channels during this change window. There is no impact to service however, we will be working with the team to identify genuine alerts and restore our monitoring as quickly as possible.

Please contact the Service Team if you need any further information or have any questions at: GDL_CustomerPlatformServiceTeam@coop.co.uk

Posted: Tue, 28 Jun 2022 10:00:00 +0000