t-mac Technologies Limited - t-mac

t-mac Support Centre

Speak to us at 0330 303 3302 | Manage Support Tickets Online | Online Guides

Snapshot Processing Delays

Notification Comments

4 comments

    We've made a change which will mean that any new snapshot data received from devices to the My.t-mac platform should be processed and become available as expected normally.

    You will see data gaps between the latest data and data received earlier on this morning, until the affected data queue has been processed.

    Please accept our apologies for this data processing delay.

    We would like to apologise for the extended recovery time regarding the snapshot data from earlier this morning.

    The affected data queue is now being processed correctly with the snapshot data now becoming accessible within the MY.t-mac platform. The time for all delayed data to become available will depend on the frequency of how often the t-mac device collects data.

    We will continue to monitor the situation and provide further updates in due course.

    All snapshot data from this morning has now been processed.

    We would like to apologise again for any inconvenience caused by this issue. If you have any outstanding issues or queries relating to this please raise a new service request to contact the support team.

    We've continually monitored the affected data queue from yesterday’s incident and can confirm that this remains stable and functioning as normal.

    After our investigates we have:

    - Reviewed our monitoring system. This issue was detected however a critical notification was not generated alerting our teams to an issue, this was discovered during a routine health check by our teams, this meant that the gap between the initial event and notification was greater than we would normally expect. We have addressed this issue by engaging with the vendor of our monitoring system to ensure that this alert is triggered, we've already increased our routine health checks against this affected queue.

    - We're continuing to investigate the root cause of this incident to enable us to put mitigations in place if this event was to occur again. In addition we are reviewing system logs to understand as to why there was a storage issue which only affected this data queue and what measures can put in place to prevent a reoccurrence.

    - As part of this resolution, our teams had to restore service through a process that has not been performed within the live production environment before, extending recovery times. We've updated our internal procedures to reflect this, therefore allowing for a swifter resolution if this incident arises again the future.

    This incident will now be marked as resolved reflecting that the affected services has remained stable and there is no on-going impact.

    If you are experiencing any issues still, or have any queries please raise a service request.

Was this notification helpful?
0 out of 0 found this helpful