Metrics Not Collected From July 4th Through July 7th
Incident Report for App47, Inc.
Postmortem

Dear Customer,

For the past couple of weeks we’ve been hard at work improving how we process data received by embedded agents. We finally got the release out last week starting on July 1st and concluding on July 4th.

The good news is that we finished the release and we are in a better position to both scale as well as add new features. Although you didn’t see any direct changes to the UI, the product is in a better position to move forward after the upgrade.

You guessed it, there is some bad news. One of the final steps on July 4th was missed by the deployment team. The result of this missed step was that data sent by the agent was not processed from approximately the afternoon of July 4th to the morning of July 8th, times in EST. The missing flow of data was not noticed by the team until a customer notified us on Monday morning.

Going forward we have made three changes:

  1. Implemented the final step that was missed; data is being recorded and we have continued monitoring this.
  2. Modified the code to fail with a warning instead of gracefully failing with no warning. This leaves the data on the agent until we can fix a future issue if it happens.
  3. We’ve always had high water alerts for the amount of data to be processed. We have now also implemented low water alerts to detect this sooner next time.

We value each of our customers and the trust the give us to handle their data at all times. We sincerely apologize for this event and will double our efforts going forward to continue to provide the service that you expect.

Posted Jul 10, 2019 - 10:23 GMT-04:00

Resolved
For the past couple of weeks we’ve been hard at work improving how we process data received by embedded agents. We finally got the release out last week starting on July 1st and concluding on July 4th.
Posted Jul 04, 2019 - 10:21 GMT-04:00