Resolved -
All systems have now fully recovered, and normal operations have resumed.
Summary of Resolution:
• Data Processing: Fully restored — all processing pipelines are stable.
• Trigger Campaigns: Operating normally with no further issues observed.
• Mass Campaigns: All previously delayed and mass-triggered campaigns have been successfully processed and delivered.
• Mass Campaign Targeting: Databricks synchronization remains healthy and up to date.
Next Steps:
• Monitoring will continue to ensure stability across all downstream services, including webhook executions and campaign reporting.
• No further impact to campaign operations is expected.
Oct 20, 17:55 PDT
Monitoring -
Current Status: Service Recovery in Progress
We’re pleased to share that our data processing systems have now recovered, and campaign operations are returning to normal.
Current Impact and Status:
- Data Processing: Fully recovered.
- Trigger Campaigns: Operating normally.
- Mass Campaigns: All Mass Gifting campaigns scheduled today are processing and expected to complete within the next 2 hours.
- Mass Campaign Targeting: Databricks lag has been cleared; synchronization is up to date.
What’s Changed:
- We have unpaused all new campaign segment executions.
- Campaign delivery and targeting are stabilizing, though some downstream tasks (e.g., webhooks) may experience minor delays as the backlog clears.
What this means for you:
- Campaigns are now running again.
- You may see slight processing delays while the system completes recovery.
We’ll continue to monitor closely and will share a final update once all campaign queues are fully processed and normal operations are fully restored.
Oct 20, 15:24 PDT
Update -
We are continuing to monitor AWS recovery efforts related to the ongoing US-East-1 outage. The issue is currently causing partial service degradation and intermittent errors across some components of our platform.
Customers may experience slower response times or brief periods of unavailability while AWS services stabilize.
Our engineering team is actively monitoring system health and will proceed with controlled service restarts once conditions in the affected region are fully stable. We will continue to provide updates as we receive more information from AWS.
Thank you for your patience and understanding as we work to minimize impact and restore full service availability.
Oct 20, 13:59 PDT
Update -
We are continuing to monitor AWS recovery efforts related to the ongoing US-East-1 outage. The issue is currently causing partial service degradation and intermittent errors across some components of our platform.
Customers may experience slower response times or brief periods of unavailability while AWS services stabilize.
Our engineering team is actively monitoring system health and will proceed with controlled service restarts once conditions in the affected region are fully stable. We will continue to provide updates as we receive more information from AWS.
Thank you for your patience and understanding as we work to minimize impact and restore full service availability.
Oct 20, 13:09 PDT
Update -
We are continuing to monitor AWS recovery efforts related to the ongoing US-East-1 outage. The issue is currently causing partial service degradation and intermittent errors across some components of our platform.
Customers may experience slower response times or brief periods of unavailability while AWS services stabilize.
Our engineering team is actively monitoring system health and will proceed with controlled service restarts once conditions in the affected region are fully stable. We will continue to provide updates as we receive more information from AWS.
Thank you for your patience and understanding as we work to minimize impact and restore full service availability.
Oct 20, 12:25 PDT
Update -
We are continuing to monitor AWS recovery efforts related to the ongoing US-East-1 outage. The issue is currently causing partial service degradation and intermittent errors across some components of our platform.
Customers may experience slower response times or brief periods of unavailability while AWS services stabilize.
Our engineering team is actively monitoring system health and will proceed with controlled service restarts once conditions in the affected region are fully stable. We will continue to provide updates as we receive more information from AWS.
Thank you for your patience and understanding as we work to minimize impact and restore full service availability.
Oct 20, 11:48 PDT
Update -
We are continuing to monitor AWS recovery efforts related to the ongoing US-East-1 outage. The issue is currently causing partial service degradation and intermittent errors across some components of our platform.
Customers may experience slower response times or brief periods of unavailability while AWS services stabilize.
Our engineering team is actively monitoring system health and will proceed with controlled service restarts once conditions in the affected region are fully stable. We will continue to provide updates as we receive more information from AWS.
Thank you for your patience and understanding as we work to minimize impact and restore full service availability.
Oct 20, 10:44 PDT
Update -
We are continuing to monitor AWS recovery efforts related to the ongoing US-East-1 outage. The issue is currently causing partial service degradation and intermittent errors across some components of our platform.
Customers may experience slower response times or brief periods of unavailability while AWS services stabilize.
Our engineering team is actively monitoring system health and will proceed with controlled service restarts once conditions in the affected region are fully stable. We will continue to provide updates as we receive more information from AWS.
Thank you for your patience and understanding as we work to minimize impact and restore full service availability.
Oct 20, 10:05 PDT
Update -
We are continuing to monitor AWS recovery efforts related to the ongoing US-East-1 outage. The issue is currently causing partial service degradation and intermittent errors across some components of our platform.
Customers may experience slower response times or brief periods of unavailability while AWS services stabilize.
Our engineering team is actively monitoring system health and will proceed with controlled service restarts once conditions in the affected region are fully stable. We will continue to provide updates as we receive more information from AWS.
Thank you for your patience and understanding as we work to minimize impact and restore full service availability.
Oct 20, 09:06 PDT
Update -
We are monitoring AWS recovery and will proceed with restart once conditions are stable.
Oct 20, 08:29 PDT
Update -
We are monitoring AWS recovery and will proceed with restart once conditions are stable.
Oct 20, 07:53 PDT
Update -
We are monitoring AWS recovery and will proceed with restart once conditions are stable.
Oct 20, 07:00 PDT
Update -
Current Status: Processing Delays
We are currently experiencing delays due to an ongoing issue impacting our Data processing. As a result, campaign operations are affected as follows:
Mass Campaigns: Delays are expected in processing and delivery.
Trigger Campaigns: There is a potential risk of targeting incorrect users.
Mass Campaign Targeting: Synchronization delays with Data processing.
What we’re doing:
We have temporarily paused all new campaign segment executions. Campaigns not already in progress will not run until we have fully recovered and cleared the current processing queue.
What this means for you:
New campaigns are on hold while we work to resolve the underlying AWS-related issue.
Once the system is stable and the existing queue is processed, we will resume campaign executions.
We’ll continue to provide updates here as we make progress.
Oct 20, 06:10 PDT
Identified -
The issue has been identified and a fix is being implemented.
Oct 20, 04:03 PDT
Update -
We are experiencing intermittent downtime and errors due to cloud service provider.
Oct 20, 01:27 PDT
Investigating -
We are currently experiencing a partial outage due to high error rates in our APIs. This issue is impacting some of our customers and may affect the performance of certain services.Our technical teams are actively investigating the problem and working diligently to resolve it as quickly as possible.
Customer Impact:
Possible Failed API requests across Punchh API’s
• Timeouts or errors displayed in the mobile app when navigating
• to different pages
• Timeouts or errors displayed when managing loyalty profile or
• rewards online
• Timeouts or errors for loyalty functionality at the POS
• Timeouts or errors from Punchh Platform Functions API
• Failed Redemptions
• Failed Checkins
• Failed Guest Lookups
• Failed signups
We apologize for any inconvenience this may cause and appreciate your patience. We will provide regular updates as we make progress. Thank you for your understanding and continued support.
Oct 20, 01:15 PDT