tag:status.airbrake.io,2005:/historyAirbrake Status - Incident History2024-03-27T19:52:22-07:00Airbraketag:status.airbrake.io,2005:Incident/189837042023-11-02T16:55:42-07:002023-11-02T16:55:42-07:00Email notifications for alerts have been disabled<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>16:55</var> PDT</small><br><strong>Resolved</strong> - This issue has been resolved. Our emails have been flowing for the past day and our sign-up page is live again. We apologize for any inconvenience. If you have any questions, please reach us at support@airbrake.io.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>12:04</var> PDT</small><br><strong>Monitoring</strong> - Updating the status to "monitoring" as alert emails are fixed. We will update the status to "resolved" when the new sign-up page is live.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>11:47</var> PDT</small><br><strong>Update</strong> - Our email provider has reenabled our alerts domain. All email notifications should be operational again. We still have our sign-up page disabled to implement an additional verification measure and will close this incident when the new sign-up page is live. If you have any questions, please reach us at support@airbrake.io.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>15:31</var> PDT</small><br><strong>Identified</strong> - Our alerts notifications have been disabled. Digests and other notification types (like Slack) are unaffected. We are working with our email service provider to reenable our alerts domain and will follow up as there is more information. Reach us at support@airbrake.io if you have any questions.</p>tag:status.airbrake.io,2005:Incident/189260832023-10-25T12:23:32-07:002023-10-25T12:23:32-07:00Email notifications not delivering<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>12:23</var> PDT</small><br><strong>Resolved</strong> - This issue has been resolved. Emails are being delivered again. Reach out to support@airbrake.io if you have any questions.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>10:42</var> PDT</small><br><strong>Identified</strong> - We have identified an issue with delivering email alerts. We are investigating the cause of the disruption and will report more as we gather more info. Errors are still being recorded and are being delivered to your error dashboards. Integration alerts are also unaffected.</p>tag:status.airbrake.io,2005:Incident/172234972023-05-12T09:11:29-07:002023-05-12T09:11:30-07:00Processing delay for Errors and APM for some customers<p><small>May <var data-var='date'>12</var>, <var data-var='time'>09:11</var> PDT</small><br><strong>Resolved</strong> - We apologize for any inconvenience this caused. The incident has been resolved and normal operations have resumed.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>16:34</var> PDT</small><br><strong>Identified</strong> - We have identified an issue affecting a certain subset of customers and are currently working on remediation. You may notice a delay in Error processing. We'll keep this issue updated. Please reach out to support@airbrake.io if you have further questions.</p>tag:status.airbrake.io,2005:Incident/165125202023-03-15T15:25:43-07:002023-03-15T15:25:44-07:00Delay in error processing<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>15:25</var> PDT</small><br><strong>Resolved</strong> - The backlog in error notifications has been processed. Users should get error emails and see errors show up on their dashboards without delays again. We apologize for any inconvenience. If you have any questions, please reach us at support@airbrake.io.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>09:38</var> PDT</small><br><strong>Identified</strong> - We have identified the issue and are currently working to process the backlog of notifications. We will continue to monitor the situation and provide updates.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>07:53</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating a delay in error processing. Errors on your error dashboard and error email notifications might take longer to appear than usual. We will share updates as we make progress in addressing this issue. If you have any questions or concerns, please reach us at support@airbrake.io. We apologize for any inconvenience.</p>tag:status.airbrake.io,2005:Incident/157323942022-12-28T17:13:23-08:002022-12-28T17:13:23-08:00Increased Errors<p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>17:13</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved. <br /><br />Services should be back to normal.</p><p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>12:41</var> PST</small><br><strong>Investigating</strong> - We are currently investigating increased errors.</p>tag:status.airbrake.io,2005:Incident/109017402022-08-24T12:25:30-07:002022-08-24T12:25:31-07:00Increased queue size, Delayed notifications<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>12:25</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>12:23</var> PDT</small><br><strong>Update</strong> - Queues are back to normal and notifications are once against being delivered.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>12:22</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>09:51</var> PDT</small><br><strong>Identified</strong> - We've identified an issue with our queue. Notification delivery is currently delayed.
<br />Thanks for your patience and sorry for any inconvenience this causes.
<br />If you have any issues or questions please email support@airbrake.io</p>tag:status.airbrake.io,2005:Incident/98492412022-04-25T04:13:23-07:002022-04-25T04:13:23-07:00Airbrake web app temporarily unavailable<p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>04:13</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved, the Airbrake web app is now working as expected.<br />If you have any issues or questions please email support@airbrake.io</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>03:53</var> PDT</small><br><strong>Investigating</strong> - Some users may not be able to access the Airbrake web app.<br />We have all hands on deck troubleshooting this dashboard outage and look forward to getting it back online ASAP. <br />Thanks for your patience and sorry for any inconvenience this causes. <br />If you have any issues or questions please email support@airbrake.io</p>tag:status.airbrake.io,2005:Incident/84681512021-11-11T06:53:39-08:002021-11-11T06:53:40-08:00Degraded dashboard experience<p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>06:53</var> PST</small><br><strong>Resolved</strong> - The cause of the incident has been identified and resolved. Systems have returned to their operational states.</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>06:12</var> PST</small><br><strong>Investigating</strong> - Some users may not be able to load error, performance, or deploy data in their dashboard. <br />The API is accepting errors as usual and you should be able to continue troubleshooting errors as soon as the dashboard has been restored. <br /><br />Thanks for your patience and sorry for any inconvenience this causes. <br />If you have any issues or questions please email support@airbrake.io.</p>tag:status.airbrake.io,2005:Incident/81387052021-10-02T14:07:16-07:002021-10-02T14:07:17-07:00Degraded performance<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>14:07</var> PDT</small><br><strong>Resolved</strong> - The backlog of notice and apm data has been processed and shard performance has stabilized. Please reach out to support (support@airbrake.io) with further questions.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>13:53</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring while the backlog of notice data / APM events are being processed.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>13:45</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>13:39</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>13:36</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating degraded performance on a database shard. Some customers may experience delays processing error and APM data.</p>tag:status.airbrake.io,2005:Incident/53886802020-10-22T08:30:06-07:002020-10-22T08:30:06-07:00Error dashboard temporarily unavailable<p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>08:30</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>07:57</var> PDT</small><br><strong>Investigating</strong> - Some users may not be able to access the error dashboard.
<br />The API is accepting errors as usual and you should be able to continue troubleshooting your errors as soon as the dashboard has been restored.
<br />We have all hands on deck troubleshooting this dashboard outage and look forward to getting it back online ASAP.
<br />Thanks for your patience and sorry for any inconvenience this causes.
<br />If you have any issues or questions please email support@airbrake.io</p>tag:status.airbrake.io,2005:Incident/44584642020-06-25T21:37:56-07:002020-06-25T21:37:57-07:00Notice and APM processing delay<p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>21:37</var> PDT</small><br><strong>Resolved</strong> - We've applied a change that relieved the load on our production database that was causing a delay in processing performance data. Delays in application performance should be resolving. If you continue to experience any issues, please let us know at support@airbrake.io.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>15:51</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>14:33</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate the issue causing a delay in processing performance data. Customer's project dashboards and email notifications should be returning to normal. If you experience any issues or have any questions, reach us at support@airbrake.io</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>09:56</var> PDT</small><br><strong>Identified</strong> - We have identified an issue where for some requests, our API is failing due to data caching errors. Affected users might experience their error dashboards failing to load and a delay in email notifications. If you have any questions or concerns, reach out to support@airbrake.io.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>08:51</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate an issue.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>08:02</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating API errors which are affecting some projects.</p>tag:status.airbrake.io,2005:Incident/44015902020-06-18T14:27:06-07:002020-06-18T14:27:07-07:00Inbound API errors<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>14:27</var> PDT</small><br><strong>Resolved</strong> - The overloaded ingress server which caused issues with deploy and error tracking was scaled up and the issues have been resolved. If you have any questions, please reach out to us at support@airbrake.io.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>12:31</var> PDT</small><br><strong>Monitoring</strong> - All systems are back to normal, we will continue to monitor the situation. If you experience any issues sending errors or tracking deploys for your project, please let us know support@airbrake.io</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>12:00</var> PDT</small><br><strong>Update</strong> - We've received reports of issues with tracking deploys, we are investigating this and believe it to be a related symptom.<br />If you are experiencing this issue please let us know support@airbrake.io, we apologize for any inconvenience this causes.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>11:53</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>11:48</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating API errors which are affecting some projects.</p>tag:status.airbrake.io,2005:Incident/40849032020-05-12T12:01:36-07:002020-05-12T12:01:38-07:00Processing delays and API errors affecting some projects<p><small>May <var data-var='date'>12</var>, <var data-var='time'>12:01</var> PDT</small><br><strong>Resolved</strong> - The issue causing the API errors and delay in error notice processing has been resolved. The backlog has been processed and systems are operating normally. We apologize for the inconvenience. Please reach us at support@airbrake.io if you have any questions.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>07:46</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results. Our APIs are back to normal and there are no more processing delays. Please contact the support at support@airbrake.io if you have any questions.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>21:59</var> PDT</small><br><strong>Identified</strong> - We have identified the issue causing API errors and processing delays which is affecting some projects. We are currently working to mitigate the outage in processing and are restarting core processing components. We apologize for the inconvenience. If you have any questions or concerns, please reach us at support@airbrake.io.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>05:58</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating API errors and processing delays which are affecting some projects. Some errors will be delayed but not lost. Sorry for the inconvenience.</p>tag:status.airbrake.io,2005:Incident/31368522019-10-23T23:00:41-07:002019-10-23T23:00:42-07:00DB Upgrades<p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>23:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>22:01</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>14:32</var> PDT</small><br><strong>Scheduled</strong> - At approximately 22:00 PDT Airbrake will undergo scheduled maintenance to upgrade its DB layer.<br /><br />How will you be affected?<br /><br />- The Airbrake UI will be unavailable for a period of up to 5 minutes<br />- Error processing may be delayed<br />- Notifications of errors won't be dropped but will be delayed<br /><br />We appreciate your patience as we continue to improve the reliability of our service.<br /><br />Please reach out to us at support@airbrake.io if you have any questions.</p>tag:status.airbrake.io,2005:Incident/30862142019-10-09T15:03:08-07:002019-10-09T15:03:09-07:00Delay in error processing and dashboard timeouts<p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>15:03</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved, you should no longer see any delay in error processing or dashboard loading. No error data was lost during this incident, only delayed. We apologize for any inconvenience this has caused! If you have issues or questions, please email support@airbrake.io</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>14:48</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>14:13</var> PDT</small><br><strong>Investigating</strong> - We’re experiencing delays in error processing and some customers may experience timeouts when loading dashboards. No error data is being lost and we are actively working to resolve this issue.<br /><br />If you have any questions or issues, please contact us at support@airbrake.io.</p>tag:status.airbrake.io,2005:Incident/27823052019-07-29T15:01:47-07:002019-07-29T15:01:48-07:00Delay in error processing<p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>15:01</var> PDT</small><br><strong>Resolved</strong> - All systems are back to normal and the backlog of data has been fully processed.<br /><br />Customers impacted by the delay should see errors again in their dashboard.<br /><br />If you have any questions or issues, please contact us at support@airbrake.io.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>14:31</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.<br /><br />For customers experiencing a delay the back log of errors is currently being processed.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>13:15</var> PDT</small><br><strong>Identified</strong> - We’re experiencing periodic delays in error processing. Some customers may also experience an empty error dashboard. No error data is being lost and we are actively working to resolve this issue.<br /><br />If you have any questions or issues, please contact us at support@airbrake.io.</p>tag:status.airbrake.io,2005:Incident/24857762019-06-03T06:52:22-07:002019-06-03T06:52:23-07:00Delay in error processing<p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>06:52</var> PDT</small><br><strong>Resolved</strong> - All systems are back to normal and the backlog of data has been fully processed. Some errors that occurred during the last (approximately) 12 hours will have been late to appear on your dashboard and to raise notifications.<br /><br />During this incident, we had some large delays in processing errors resulting in out-of-date dashboard data and late error notifications. This is not acceptable service and for this, we're very sorry. We're working hard to learn and improve from what happened.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>04:32</var> PDT</small><br><strong>Monitoring</strong> - The fix was successful and is complete, we expect to be fully restored to normal operation very soon. We are processing a large backlog of error data and as a result, errors that occurred in your projects during the last ~12 hours are likely to start appearing and raising notifications. We are extremely sorry for this delay and unsatisfactory service. We'll close with a more detailed postmortem, soon.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>03:12</var> PDT</small><br><strong>Identified</strong> - A fix is being implemented. We hope to be fully operational in approximately 1 hour, but it's difficult to predict at this time. We are very sorry for this major disruption. We are working hard to resolve it quickly. If you have any questions or concerns, you can reach us at support@airbrake.io.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>01:10</var> PDT</small><br><strong>Update</strong> - Some customers may also experience an empty error dashboard. We're working hard to fix this as soon as possible.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>01:08</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>21:25</var> PDT</small><br><strong>Investigating</strong> - Some accounts may be experiencing delays in error processing. We’re actively investigating and will post an update once we learn more. If you have any questions, please contact us at support@airbrake.io.</p>tag:status.airbrake.io,2005:Incident/20060682018-10-31T07:46:39-07:002018-10-31T07:46:39-07:00Delay in error processing<p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>07:46</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>07:17</var> PDT</small><br><strong>Identified</strong> - We’re experiencing some delays in error processing and we’re actively working on a fix. If you have any questions, please contact us at support@airbrake.io.</p>tag:status.airbrake.io,2005:Incident/18521892018-08-08T17:16:57-07:002018-08-08T17:16:57-07:00Intermittent redirect loops<p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>17:16</var> PDT</small><br><strong>Resolved</strong> - We've identified the issue and pushed a fix. Please email us at support@airbrake.io if you have any further issues or any questions.</p><p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>16:54</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an intermittent issue causing redirect loops when trying to view a project. We apologize for the inconvenience, this issue is not affecting error processing. Please email any questions or concerns to support@airbrake.io</p>tag:status.airbrake.io,2005:Incident/18113672018-07-13T06:20:15-07:002018-07-13T06:20:15-07:00Most email notifications not being sent.<p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>06:20</var> PDT</small><br><strong>Resolved</strong> - Emails are back to full flow but. Our email provider is monitoring their fix. Many email notifications are lost. We're very sorry for this inconvenience. We'll continue to monitor for any changes.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>05:57</var> PDT</small><br><strong>Monitoring</strong> - Many (but not all) email notifications are flowing again. Our email provider has identified the problem and is working on a fix.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>05:44</var> PDT</small><br><strong>Identified</strong> - Most email notifications are not being sent. Our email provider has problems. We're monitoring and will provide updates as soon as possible. More details here http://status.mailgun.com/incidents/5jz5w33sx2qv</p>tag:status.airbrake.io,2005:Incident/17548842018-06-04T10:16:49-07:002018-06-04T10:16:50-07:00Delay in notifications<p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>10:16</var> PDT</small><br><strong>Resolved</strong> - The issue with notifications has been resolved. No errors were lost and they continue to be processed normally, however some notifications were undelivered. If you have any questions, please reach us at support@airbrake.io.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>09:28</var> PDT</small><br><strong>Investigating</strong> - We're investigating an issue causing a delay in notification delivery. Errors are being processed as normal. If you have any questions or concerns, please reach us at support@airbrake.io.</p>tag:status.airbrake.io,2005:Incident/17191022018-05-07T03:48:56-07:002018-05-07T03:48:56-07:00Delay in error processing<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>03:48</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>21:29</var> PDT</small><br><strong>Identified</strong> - We’re experiencing periodic delays in error processing. No error data is being lost and we’re actively working to speed up the error processing. If you have any questions or issues, please contact us at support@airbrake.io.</p>tag:status.airbrake.io,2005:Incident/17188342018-05-06T13:52:46-07:002018-05-06T13:52:47-07:00Delay in error processing and notifications<p><small>May <var data-var='date'> 6</var>, <var data-var='time'>13:52</var> PDT</small><br><strong>Resolved</strong> - The delay in processing errors and notifications has been resolved. The backlog in errors is being processed. If you have any questions or issues, please contact us at support@airbrake.io.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>11:30</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating a delay in error processing and notifications. Please reach out to us at support@airbrake.io if you have any questions or concerns. We will post updates as we address the issue.</p>tag:status.airbrake.io,2005:Incident/17019672018-04-23T15:20:23-07:002018-04-23T15:20:23-07:00Delay in error notifications<p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>15:20</var> PDT</small><br><strong>Resolved</strong> - The delay in error notifications has been resolved and the backlog is being processed. If you have any questions, let us know at support@airbrake.io.</p><p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>14:53</var> PDT</small><br><strong>Investigating</strong> - We're currently investigating a delay in error notifications. The error dashboard and web app are unaffected. Reach us at support@airbrake.io if you have any questions.</p>tag:status.airbrake.io,2005:Incident/16610102018-03-27T07:08:38-07:002018-03-27T07:08:39-07:00Dashboard & website unavailable, intermittently<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>07:08</var> PDT</small><br><strong>Resolved</strong> - This is resolved, sorry for the bumps. We're working on preventing future occurrences.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>06:00</var> PDT</small><br><strong>Investigating</strong> - The dashboard and website are unavailable intermittently.</p>