https://www.loomstatus.com/Loom status2024-03-29T08:37:35.049Zincident.ioLoom status page updateshttps://storage.googleapis.com/incident-io-status-page-logos/01F0CNKGFW4NZJWK8415DVWRG4/loom/8nn6tv98.pnghttps://www.loomstatus.com//incidents/01HRZKSPVR04MY3C3V0G8RC9912024-03-14T23:32:38.415ZStatus: Resolved
Our engineering team was able to locate the issue and it's been fixed. Edit by Transcript is now fully accessible again!]]>Status: Resolved
Our engineering team was able to locate the issue and it's been fixed. Edit by Transcript is now fully accessible again!]]>https://www.loomstatus.com//incidents/01HQR5EZDH7TNYAKRM09BHGH342024-02-29T20:40:12.868ZStatus: Resolved
This issue has been resolved]]>Status: Resolved
This issue has been resolved]]>https://www.loomstatus.com//incidents/01HQB8T698AAEGRC742263MHJX2024-02-23T16:21:43.044ZStatus: Resolved
This issue has been resolved as of v5.5.40 of the Chrome extension]]>Status: Resolved
This issue has been resolved as of v5.5.40 of the Chrome extension]]>https://www.loomstatus.com//incidents/01HG95H7Q79E0G1ZN249RSV08S2023-11-27T23:13:47.241ZStatus: Resolved
We have issued a fix for the editing issues. Loom should be back to normal.]]>Status: Resolved
We have issued a fix for the editing issues. Loom should be back to normal.]]>https://www.loomstatus.com//incidents/01HA79WT3R3RRRV28SKFQG76SP2023-09-13T13:59:34.095ZStatus: Resolved
We have resolved the issue with Transcript and Audio Waveform generation failing.]]>Status: Resolved
We have resolved the issue with Transcript and Audio Waveform generation failing.]]>https://www.loomstatus.com//incidents/01GYX8W59PGNJBE59EF4CF9SGK2021-12-23T07:44:07.285ZStatus: Resolved
Issues with one of our upstream dependencies have been resolved. SSO login is operational again.]]>Status: Resolved
Issues with one of our upstream dependencies have been resolved. SSO login is operational again.]]>https://www.loomstatus.com//incidents/01GYX8W5FCM9SZV6R7MRH69NJH2021-12-15T16:12:14.420ZStatus: Resolved
We are now seeing latencies go back to normal levels. Parts of Loom application experiencing high latency between 10:15 AM ET / 3:15PM UTC to 11:15 AM ET / 4:03PM UTC. We will continue to monitor and proactively update the status page in case anything changes. Thank you!
Affected components
Viewing (Operational)
Recording (Operational)
]]>Status: Resolved
We are now seeing latencies go back to normal levels. Parts of Loom application experiencing high latency between 10:15 AM ET / 3:15PM UTC to 11:15 AM ET / 4:03PM UTC. We will continue to monitor and proactively update the status page in case anything changes. Thank you!
A database schema change resulted in issues starting 7:12 PM UTC. These issues were fully resolved by 7:42 PM UTC.]]>Status: Resolved
A database schema change resulted in issues starting 7:12 PM UTC. These issues were fully resolved by 7:42 PM UTC.]]>https://www.loomstatus.com//incidents/01GYX8W5JBM7CR27XP9VMAJN152021-11-11T04:04:12.283ZStatus: Resolved
This incident has been resolved.]]>Status: Resolved
This incident has been resolved.]]>https://www.loomstatus.com//incidents/01GYX8W5M306E51242AK1B0BCF2021-11-02T23:51:51.909ZStatus: Resolved
Yesterday (9/23/21), Google shipped a security patch in their latest Chrome release which adds a new feature policy gating access to display-capture. This release breaks loomSDK recordings for users on the latest version of Chrome. (Google details Feature Policy: display-capture - Chrome Platform Status: https://chromestatus.com/feature/5144822362931200)
We have patched our server-side functionality for verifying that the user initiating a loom recording is in a supported environment to return false, so any users on latest Chrome should not be able to load the loomSDK. We have also released an updated loom-sdk package which addresses the Chrome update. This version is 2.14.2.
In order to unblock users on latest Chrome from recording with the loomSDK, partners will need to update to the 2.14.2 release.
We will continue monitoring while partners perform these upgrades.]]>Status: Resolved
Yesterday (9/23/21), Google shipped a security patch in their latest Chrome release which adds a new feature policy gating access to display-capture. This release breaks loomSDK recordings for users on the latest version of Chrome. (Google details Feature Policy: display-capture - Chrome Platform Status: https://chromestatus.com/feature/5144822362931200)
We have patched our server-side functionality for verifying that the user initiating a loom recording is in a supported environment to return false, so any users on latest Chrome should not be able to load the loomSDK. We have also released an updated loom-sdk package which addresses the Chrome update. This version is 2.14.2.
In order to unblock users on latest Chrome from recording with the loomSDK, partners will need to update to the 2.14.2 release.
We will continue monitoring while partners perform these upgrades.]]>https://www.loomstatus.com//incidents/01GYX8W5QDMJT33ACYP7S6KSPF2021-09-19T05:15:30.417ZStatus: Complete
We identified the issue and rolled out a fix. Authentications are now working as expected via the SDK. Thank you for your patience!]]>Status: Resolved
We identified the issue and rolled out a fix. Authentications are now working as expected via the SDK. Thank you for your patience!]]>https://www.loomstatus.com//incidents/01GYX8W5VFXTG1BG7EGHSY20C82021-08-31T23:00:50.966ZStatus: Resolved
The incident is now resolved. We were able to route traffic in a way that would circumvent upstream cloud provider issues. We apologies for any inconvenience and really appreciate your patience. Thank you for recording with us.
Affected components
Viewing (Operational)
Recording (Operational)
]]>Status: Resolved
The incident is now resolved. We were able to route traffic in a way that would circumvent upstream cloud provider issues. We apologies for any inconvenience and really appreciate your patience. Thank you for recording with us.
The issue has been resolved and the screenshots page is fully operational.]]>Status: Resolved
The issue has been resolved and the screenshots page is fully operational.]]>https://www.loomstatus.com//incidents/01GYX8W606KEXHHS19G175GE272021-08-02T20:06:55.835ZStatus: Resolved
All mobile push notifications systems are operational.]]>Status: Resolved
All mobile push notifications systems are operational.]]>https://www.loomstatus.com//incidents/01GYX8W61XDYFZ2BTYFZTH8JW62021-07-28T01:56:43.394ZStatus: Resolved
The issue has now been fixed and we have rolled out version 5.1.0 of the extension to all users]]>Status: Resolved
The issue has now been fixed and we have rolled out version 5.1.0 of the extension to all users]]>https://www.loomstatus.com//incidents/01GYX8W64D861F5QB70372RK942021-07-26T14:53:58.536ZStatus: Resolved
The issue regarding setting password-protected videos has been resolved. The ability to set passwords on videos now works again for all users.
Affected components
Viewing (Operational)
]]>Status: Resolved
The issue regarding setting password-protected videos has been resolved. The ability to set passwords on videos now works again for all users.
The issue has been resolved. Please upgrade your Chrome Extension to version 4.2.0 via the Chrome Web Store or restart your Google Chrome to reload Loom.
Affected components
Recording (Operational)
]]>Status: Resolved
The issue has been resolved. Please upgrade your Chrome Extension to version 4.2.0 via the Chrome Web Store or restart your Google Chrome to reload Loom.
This incident has been resolved.]]>Status: Resolved
This incident has been resolved.]]>https://www.loomstatus.com//incidents/01GYX8W6ADSX47PE8PDJ2SARPR2021-06-18T14:38:51.000ZStatus: Resolved
All systems returned to normal]]>Status: Resolved
All systems returned to normal]]>https://www.loomstatus.com//incidents/01GYX8W6C9HF3T1WPSMT9FJ6JD2021-06-16T14:59:55.236ZStatus: Resolved
All systems returned to normal]]>Status: Resolved
All systems returned to normal]]>https://www.loomstatus.com//incidents/01GYX8W6E24W3ZPK6HQFPPKD822021-06-07T13:47:52.544ZStatus: Resolved
Share button should now be restored on all video pages.]]>Status: Resolved
Share button should now be restored on all video pages.]]>https://www.loomstatus.com//incidents/01GYX8W6H512JSPGCWTE1ABT6Y2021-05-27T18:07:00.654ZStatus: Resolved
We've released a fix for the issue. We're waiting to see if it has worked.]]>Status: Resolved
We've released a fix for the issue. We're waiting to see if it has worked.]]>https://www.loomstatus.com//incidents/01GYX8W6RMZ4V9V08F53VPF9SQ2021-04-20T22:05:01.900ZStatus: Resolved
We have recovered to regular levels of requests and no issues are being reported. The incident is now resolved.
Affected components
Viewing (Operational)
]]>Status: Resolved
We have recovered to regular levels of requests and no issues are being reported. The incident is now resolved.
Between 1:12 PM EDT to 1:27 PM EDT operations across all Loom applications faced high latencies and error rates. We identified the issue to occur from a schema migration running against our primary database. We were able to detect and mitigate the issue quickly. We sincerely apologize for any inconvenience caused.]]>Status: Resolved
Between 1:12 PM EDT to 1:27 PM EDT operations across all Loom applications faced high latencies and error rates. We identified the issue to occur from a schema migration running against our primary database. We were able to detect and mitigate the issue quickly. We sincerely apologize for any inconvenience caused.]]>https://www.loomstatus.com//incidents/01GYX8W6YD59YXAVSN5V56Q5RP2020-12-12T13:59:01.205ZStatus: Complete
Between 8:30PM PST to 8:37PM PST operations across Loom app faced high latencies and error rates. We identified the issue and mitigated it quickly. We sincerely apologize for any inconvenience caused.]]>Status: Resolved
Between 8:30PM PST to 8:37PM PST operations across Loom app faced high latencies and error rates. We identified the issue and mitigated it quickly. We sincerely apologize for any inconvenience caused.]]>https://www.loomstatus.com//incidents/01GYX8W7207BWJ6M86CSTV5GP62020-08-19T21:00:07.759ZStatus: Resolved
We have resolved the incident. Recordings were not possible during this period from the Chrome extension or Windows desktop application. The cause of the incident was an expired certificate in a load balancing layer of the application which has been updated. The team has decided to move off of an old system entirely to avoid this issue from happening again.
Affected components
Recording (Operational)
]]>Status: Resolved
We have resolved the incident. Recordings were not possible during this period from the Chrome extension or Windows desktop application. The cause of the incident was an expired certificate in a load balancing layer of the application which has been updated. The team has decided to move off of an old system entirely to avoid this issue from happening again.
We experienced a slow down in our website at 6:10am - 6:25am PST. This was due to 2 large database migrations that went awry. During this incident, 2 of our recording servers went into a state where they couldn't accept reconnections for recordings. This means a healthy recording was not impacted, but we lost 20-30 recordings that needed to reconnect to our servers.
Affected components
Viewing (Operational)
Recording (Operational)
]]>Status: Resolved
We experienced a slow down in our website at 6:10am - 6:25am PST. This was due to 2 large database migrations that went awry. During this incident, 2 of our recording servers went into a state where they couldn't accept reconnections for recordings. This means a healthy recording was not impacted, but we lost 20-30 recordings that needed to reconnect to our servers.
We had complete downtime across all systems. The cause of this downtime was because of the following:
• A db lock implementation that spun indefinitely against the database and didn't release gracefully
• An unindexed, large user table scan that resulted in long lookup times when logging users in and signing them up
• The second issue exacerbating the load on the DB because of the first issue
Remediation:
• Replace our db lock implementation with one that does not spin
• Denormalize the lookup information off the user table and added indexes for faster lookup times
• Implemented this status page
• Upgrade database to latest version
• Upgraded server database runs on to double CPU and Memory
• Created dashboards internally that get us rich information about the health of our database queries so we can diagnose and address query-related issues before they impact production traffic]]>Status: Resolved
We had complete downtime across all systems. The cause of this downtime was because of the following:
• A db lock implementation that spun indefinitely against the database and didn't release gracefully
• An unindexed, large user table scan that resulted in long lookup times when logging users in and signing them up
• The second issue exacerbating the load on the DB because of the first issue
Remediation:
• Replace our db lock implementation with one that does not spin
• Denormalize the lookup information off the user table and added indexes for faster lookup times
• Implemented this status page
• Upgrade database to latest version
• Upgraded server database runs on to double CPU and Memory
• Created dashboards internally that get us rich information about the health of our database queries so we can diagnose and address query-related issues before they impact production traffic]]>