[{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000wk5cEAA"},"Id":"a9C4z000000wk5cEAA","Duration__c":0,"Log__c":"Okta is experiencing an issue where admins may be experiencing errors in utilizing Workflow designer Engineering is currently investigating the issue. We will provide an update within the next 30 minutes.","Status__c":"Resolved","Start_Date__c":"2021-11-08","End_Date__c":"2021-11-08","Last_Updated__c":"2021-11-10T17:12:55.000+0000","LastModifiedDate":"2021-11-10T17:12:55.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Small","Incident_Title__c":"Errors Invoking New Workflows, Utilizing Workflows Designer, and Hook Delays","Impacted_Cells__c":"OK1;OK2;OK3;OK4;OK6;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:6;okta.com:7;OK7;okta.com:11","Impacted_Audience__c":"Admin","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Workflows","Start_Time__c":"2021-11-08T20:46:00.000+0000","CreatedDate":"2021-11-08T20:48:43.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000wk5IEAQ"},"Id":"a9C4z000000wk5IEAQ","Duration__c":0,"Log__c":"Okta is currently experiencing an issue with third-party SMS provider issue with ATT service. We are actively investigating and will update this message with more information as soon as we have it. For those users with a secondary MFA factor enrolled, a backup factor can be used.","Status__c":"Resolved","Start_Date__c":"2021-10-27","Last_Updated__c":"2021-10-27T22:17:38.000+0000","LastModifiedDate":"2021-10-27T22:17:38.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Small","Service_Feature__c":"Third Party","Okta_Sub_Service__c":"MFA","Start_Time__c":"2021-10-27T12:42:00.000+0000","CreatedDate":"2021-10-27T12:46:29.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000wk4eEAA"},"Id":"a9C4z000000wk4eEAA","Duration__c":9,"Log__c":"Okta has experienced an intermittent issue in loading custom domains in US Cell 10. Mitigation steps were effective at 7.55 PT and resolved the issue in US Cell 10. Additional root cause information will be provided within 48 business hours.","Status__c":"Resolved","Start_Date__c":"2021-10-14","Last_Updated__c":"2021-10-15T03:27:06.000+0000","LastModifiedDate":"2021-10-15T03:27:06.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Small","Impacted_Cells__c":"okta.com:10","Impacted_Audience__c":"Admin;API Products;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform;Single Sign-On","Start_Time__c":"2021-10-15T02:48:00.000+0000","CreatedDate":"2021-10-15T02:49:23.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000wk4KEAQ"},"Id":"a9C4z000000wk4KEAQ","Duration__c":240,"Log__c":"Okta have experienced a SAML Single Sign On issue with the AWS app. We have resolved the issue across all cells. Additional root cause information will be provided within 48 business hours.","Status__c":"Resolved","Start_Date__c":"2021-10-05","Last_Updated__c":"2021-10-05T13:11:25.000+0000","LastModifiedDate":"2021-10-05T13:11:26.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"Okta is currently experiencing a SSO issue with the AWS app","Impacted_Cells__c":"EU1;Preview;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;Preview2;okta.com:8;okta.com:9;okta.com:10;okta.com:11;okta.com:12;okta.com:15;oktapreview.com:3;okta.com:14","Impacted_Audience__c":"Admin","Service_Feature__c":"Third Party","Okta_Sub_Service__c":"Single Sign-On","Start_Time__c":"2021-10-05T11:07:00.000+0000","CreatedDate":"2021-10-05T11:13:00.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000wk3CEAQ"},"Id":"a9C4z000000wk3CEAQ","Duration__c":276,"Log__c":"Okta is currently investigating increased error rates in US Cells 5, 7, 12, and 14, and Preview 3. We will update this message with more information as soon as we have it.\r\n\r\nResolved: We have mitigated all observed errors and issues in the US-West region affecting US cells 5, 7, 12, 14, EU 9, and Preview 3. Error rates have completely subsided and the service has returned to normal status, queued events are progressing and should complete in the next few hours. Additional root cause information will provided within 2 business days.","Status__c":"Resolved","Start_Date__c":"2021-08-31","Last_Updated__c":"2021-09-03T03:24:43.000+0000","LastModifiedDate":"2021-09-03T03:24:43.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta.com:5;okta.com:7;okta.com:9;okta.com:12;oktapreview.com:3;okta.com:14","Impacted_Audience__c":"Admin;API Products;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform;Access Gateway;API Services;MFA;Single Sign-On;Workflows","Start_Time__c":"2021-08-31T17:59:00.000+0000","CreatedDate":"2021-08-31T18:47:05.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000wk37EAA"},"Id":"a9C4z000000wk37EAA","Duration__c":51,"Log__c":"Resolved: We are investigating errors on OK5, we are actively working to resolve and will provide further updates.","Status__c":"Resolved","Start_Date__c":"2021-08-28","Last_Updated__c":"2021-09-01T18:05:09.000+0000","LastModifiedDate":"2021-09-01T18:05:09.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta.com:5","Impacted_Audience__c":"Admin;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2021-08-29T03:44:00.000+0000","CreatedDate":"2021-08-29T03:44:59.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000wk32EAA"},"Id":"a9C4z000000wk32EAA","Duration__c":127,"Log__c":"Resolved: We observed intermittent issues in OK12 for customers utilizing Custom URL Domain functionality. As of 8:15pm, the system appears to be recovering. Additional root cause information will be provided in 2 business days.","Status__c":"Resolved","Start_Date__c":"2021-08-25","Last_Updated__c":"2021-08-28T07:06:52.000+0000","LastModifiedDate":"2021-08-28T07:06:52.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta.com:12","Impacted_Audience__c":"Admin","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2021-08-26T03:09:00.000+0000","CreatedDate":"2021-08-26T03:09:49.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000wk2nEAA"},"Id":"a9C4z000000wk2nEAA","Duration__c":115,"Log__c":"Beginning at approximately 5:27AM PDT, customers leveraging SMS as a factor may have experienced intermittent delivery delays to U.S based numbers upon first attempt. At 7:22am Okta promoted our secondary SMS delivery provider to function as our primary provider. An RCA will be provided following full investigation.","Status__c":"Resolved","Start_Date__c":"2021-08-06","End_Date__c":"2021-08-06","Last_Updated__c":"2021-08-23T19:39:25.000+0000","LastModifiedDate":"2021-08-23T19:39:25.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"SMS MFA Delivery Delay to US Numbers","Impacted_Cells__c":"okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;okta.com:8;okta.com:9;okta.com:10;okta.com:11;okta.com:12;okta.com:15;oktapreview.com:3;okta.com:14","Impacted_Audience__c":"End user","Service_Feature__c":"Third Party","Okta_Sub_Service__c":"MFA","Start_Time__c":"2021-08-06T12:10:00.000+0000","CreatedDate":"2021-08-06T17:11:21.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000wk2sEAA"},"Id":"a9C4z000000wk2sEAA","Duration__c":10,"Log__c":"Resolved: An issue impacting access to US Cell 10 was resolved. During this event, End-Users and Admins may have experienced issues logging into their Okta tenant. Additional root cause information will be provided within 48 business hours.","Status__c":"Resolved","Start_Date__c":"2021-08-09","Last_Updated__c":"2021-08-12T02:33:47.000+0000","LastModifiedDate":"2021-08-12T02:33:47.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Small","Incident_Title__c":"Unable to Access ORG","Impacted_Cells__c":"okta.com:10","Impacted_Audience__c":"Admin;API Products;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform;API Services;MFA;Single Sign-On","Start_Time__c":"2021-08-09T21:40:00.000+0000","CreatedDate":"2021-08-09T21:41:52.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000oLnJEAU"},"Id":"a9C4z000000oLnJEAU","Duration__c":40,"Log__c":"Resolved: An issue impacting access to ORGs has been resolved. An RCA will be provided within 48 business hours.","Status__c":"Resolved","Start_Date__c":"2021-06-23","Last_Updated__c":"2021-06-25T16:20:43.000+0000","LastModifiedDate":"2021-06-25T16:20:43.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Medium","Impacted_Cells__c":"okta.com:6","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform;API Services;MFA;Single Sign-On","Start_Time__c":"2021-06-24T04:07:00.000+0000","CreatedDate":"2021-06-24T04:10:58.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000oLmzEAE"},"Id":"a9C4z000000oLmzEAE","Duration__c":51,"Log__c":"Resolved: An issue impacting workflows dashboard access and flow execution has been resolved. An RCA will be provided within 48 business hours.","Status__c":"Resolved","Start_Date__c":"2021-06-11","Last_Updated__c":"2021-06-15T21:42:48.000+0000","LastModifiedDate":"2021-06-15T21:42:48.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"Workflows Not Running and Issues Loading Console","Impacted_Cells__c":"okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:6;okta.com:7;okta.com:11","Impacted_Audience__c":"Admin;API Products","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Workflows","Start_Time__c":"2021-06-11T17:57:00.000+0000","CreatedDate":"2021-06-11T18:02:39.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000oLmVEAU"},"Id":"a9C4z000000oLmVEAU","Duration__c":120,"Log__c":"Resolved: The issue involving the RingCentral OIN Application has been resolved. We are continuing to monitor. An RCA will be provided within 48 business hours.","Status__c":"Resolved","Start_Date__c":"2021-05-26","Last_Updated__c":"2021-05-27T02:40:09.000+0000","LastModifiedDate":"2021-05-27T02:40:09.000+0000","Category__c":"Minor Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"Failures Provisioning to RingCentral OIN Application","Impacted_Cells__c":"EU1;OK1;OK2;OK3;OK4;OK5;OK6;Preview;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;OK7;Preview2;okta.com:8;okta.com:9;okta.com:10;okta.com:11;okta.com:12;okta.com:15;oktapreview.com:3","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2021-05-27T00:20:00.000+0000","CreatedDate":"2021-05-27T00:38:08.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000oLmQEAU"},"Id":"a9C4z000000oLmQEAU","Duration__c":44,"Log__c":"End users authenticating to the Okta service may have experienced intermittent errors and increased latency between 11:04 and 11:47 am PDT. Okta will provide an RCA within the next 48 hours.","Status__c":"Resolved","Start_Date__c":"2021-05-25","End_Date__c":"2021-05-25","Last_Updated__c":"2021-05-25T23:29:21.000+0000","LastModifiedDate":"2021-05-25T23:29:21.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Medium","Incident_Title__c":"Intermittent Errors and Increased Latency on Authentication","Impacted_Cells__c":"okta.com:6","Impacted_Audience__c":"Admin;API Products;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform;API Services;MFA;Single Sign-On","Start_Time__c":"2021-05-25T18:04:00.000+0000","CreatedDate":"2021-05-25T23:29:21.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000oLm1EAE"},"Id":"a9C4z000000oLm1EAE","Duration__c":61,"Log__c":"Okta experienced a service disruption in OK6 starting at approximately 10:44 am PDT. Users would have experienced intermittent access, slowness and failures. As of 11:45 am PDT, OKTA services were fully restored. Okta will be providing a Root Cause Analysis within the next 48 hours.","Status__c":"Resolved","Start_Date__c":"2021-05-13","Last_Updated__c":"2021-05-15T14:23:43.000+0000","LastModifiedDate":"2021-05-15T14:23:43.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta.com:6","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform;Access Gateway;Advanced Server Access;API Services;MFA;Single Sign-On;Workflows","Start_Time__c":"2021-05-13T18:30:00.000+0000","CreatedDate":"2021-05-13T18:32:54.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000oLkyEAE"},"Id":"a9C4z000000oLkyEAE","Duration__c":76,"Log__c":"Resolved: Microsoft has reported that Microsoft DNS service is restored.  Okta Services related to Microsoft Office 365 and Azure are working as expected.  We are working with the service provider to understand the root cause of this service disruption and will post the RCA to status.okta.com within 48 hours.","Status__c":"Resolved","Start_Date__c":"2021-04-01","Last_Updated__c":"2021-04-05T13:10:56.000+0000","LastModifiedDate":"2021-04-05T13:10:56.000+0000","Category__c":"Feature Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Service_Feature__c":"Third Party","Start_Time__c":"2021-04-01T22:14:00.000+0000","CreatedDate":"2021-04-01T22:17:07.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000oLkeEAE"},"Id":"a9C4z000000oLkeEAE","Duration__c":1479,"Log__c":"Microsoft 365 Service reporting Azure authentication down per [status.office.com](http://status.office.com). Microsoft is actively investigating and we will update this message with more information as soon as we have it.\r\n\r\nResolved: Microsoft has reported that Azure AD service is restored. All Okta Services related to Microsoft Office 365 and Azure are working as expected with the exception of some connections to Azure Active Directory Graph API for Okta Provisioning service which we expect to be slower to recover. We will continue to monitor residual effects for a limited number of customers. We are working with the service provider to investigate the root cause of this service disruption and will post the RCA to [status.okta.com](http://status.okta.com.com) within 48 hours.","Status__c":"Resolved","Start_Date__c":"2021-03-15","Last_Updated__c":"2021-03-22T16:31:00.000+0000","LastModifiedDate":"2021-03-22T16:31:00.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"EU1;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;okta.com:8;okta.com:9;okta.com:10;okta.com:11;okta.com:12;okta.com:15;oktapreview.com:3","Impacted_Audience__c":"Admin;End user","Service_Feature__c":"Third Party","Start_Time__c":"2021-03-15T20:49:00.000+0000","CreatedDate":"2021-03-15T20:52:55.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C4z000000oLkPEAU"},"Id":"a9C4z000000oLkPEAU","Duration__c":160,"Log__c":"Okta is currently experiencing intermittent issues with external network connectivity in the OK8 (APAC) region. Depending on location, users may not be able to access our service. We are actively investigating and will update this message with more information.\r\n_________\r\n\r\nResolved: Okta experienced a service disruption in the OK8 cell (APAC) where customers may have experienced intermittent network connectivity due to a service provider issue. Service has returned to normal as of 11:30 AM PST. We are working with the service provider to investigate the root cause of this service disruption and will post the RCA to https://status.okta.com within 48 hours.","Status__c":"Resolved","Start_Date__c":"2021-03-07","End_Date__c":"2021-03-07","Last_Updated__c":"2021-03-09T19:02:35.000+0000","LastModifiedDate":"2021-03-09T19:02:35.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"Okta Service Outage For OK8 (APAC)","Impacted_Cells__c":"okta.com:8","Impacted_Audience__c":"Admin;API Products;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2021-03-07T17:42:00.000+0000","CreatedDate":"2021-03-07T17:46:32.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008Ph5UAE"},"Id":"a9C1Y0000008Ph5UAE","Duration__c":2369,"Log__c":"Resolved: An issue resulting in intermittent Azure Active Directory Graph API provisioning errors has been resolved. Root Cause Analysis (RCA) will be made available within 2 business days.","Status__c":"Resolved","Start_Date__c":"2021-02-16","Last_Updated__c":"2021-02-21T20:09:59.000+0000","LastModifiedDate":"2021-02-21T20:09:59.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"Intermittent Graph API failures when provisioning users to Office 365","Impacted_Cells__c":"EU1;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;okta.com:8;okta.com:9;okta.com:10;okta.com:11;okta.com:12;okta.com:15;oktapreview.com:3","Impacted_Audience__c":"Admin;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2021-02-16T19:56:00.000+0000","CreatedDate":"2021-02-16T20:00:22.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008PhPUAU"},"Id":"a9C1Y0000008PhPUAU","Duration__c":0,"Log__c":"Customers are reporting intermittent User Interface (UI) issues accessing their Okta cell 5 orgs. We are actively working to resolve the issue.\r\n\r\n==\r\n\r\nResolved: All authentications in Okta cell 5 are now succeeding. We are continuing to look into a remaining cosmetic issue where certain static images (such as logos and background images) aren’t loading properly but this is not impacting functionality. Root cause analysis will be completed within the next 2 business days.","Status__c":"Resolved","Start_Date__c":"2021-02-20","Last_Updated__c":"2021-02-20T23:28:07.000+0000","LastModifiedDate":"2021-02-20T23:28:07.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Small","Impacted_Cells__c":"okta.com:5","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2021-02-20T22:17:00.000+0000","CreatedDate":"2021-02-20T22:18:36.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008PggUAE"},"Id":"a9C1Y0000008PggUAE","Duration__c":806,"Log__c":"Jobs were reported to be delayed across all Okta Cells, except US Cell 5. Okta has fixed the root cause of the issue related to the jobs being delayed. All impacted services are now restored and fully functional. Remaining job backlogs are either completely caught up, or will return to normal operation in due course.  A Root Cause Analysis (RCA) will be made available within 2 business days. \r\n\r\n=============================\r\n\r\nResolved - Jobs are delayed across all Okta Cells, except US Cell 5. Okta has identified a solution to resolve the use and is being validated in the Okta preview Cell.","Status__c":"Resolved","Start_Date__c":"2021-02-03","Last_Updated__c":"2021-02-06T02:30:13.000+0000","LastModifiedDate":"2021-02-06T02:30:13.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"AD Import Job start are delayed","Impacted_Cells__c":"EU1;OK1;OK2;OK3;OK4;OK5;OK6;Preview;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;OK7;Preview2;okta.com:8;okta.com:9;okta.com:10;okta.com:11;okta.com:12;okta.com:15;oktapreview.com:3","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2021-02-03T22:07:00.000+0000","CreatedDate":"2021-02-03T22:09:18.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008PgbUAE"},"Id":"a9C1Y0000008PgbUAE","Duration__c":86,"Log__c":"Okta has resolved the Office 365 login issue. The issue was limited to users with thick clients and using certain Edge browser versions. The issue impacted all cells, except US Cell 5. Root Cause Analysis (RCA) will be made available within 2 business days.\r\n\r\n\r\n\r\n============================================\r\n\r\n\r\nCustomers have reported issues logging to Office365. The issue is limited to thick clients and certain browser versions. This may impact all cells, except OK5. We  are actively investigating and will provide additional updates shortly.","Status__c":"Resolved","Start_Date__c":"2021-02-03","Last_Updated__c":"2021-02-05T23:13:38.000+0000","LastModifiedDate":"2021-02-05T23:13:38.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"Office 365 Client issue","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Single Sign-On","Start_Time__c":"2021-02-03T16:09:00.000+0000","CreatedDate":"2021-02-03T16:19:08.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008PgWUAU"},"Id":"a9C1Y0000008PgWUAU","Duration__c":210,"Log__c":"Okta has resolved the issue related to Okta LDAP interface in the US Cell 5. All services for Customers are now restored and fully functional. Root Case Analysis (RCA) will be made available within 2 business days.\r\n\r\n ======================\r\n\r\nCustomers in US Cell 5 are experiencing trouble connecting to Okta LDAP interface. We are actively investigating and will provide additional updates shortly.","Status__c":"Resolved","Start_Date__c":"2021-02-02","Last_Updated__c":"2021-02-05T19:15:59.000+0000","LastModifiedDate":"2021-02-05T19:15:59.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"LDAPi Issue impacting Cell5","Impacted_Cells__c":"okta.com:5","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2021-02-03T00:31:00.000+0000","CreatedDate":"2021-02-03T00:33:39.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008PfEUAU"},"Id":"a9C1Y0000008PfEUAU","Duration__c":34,"Log__c":"Okta has experienced an issue with the service. Increase in error rates accessing EU1 cell.\r\n\r\n=======\r\n\r\nResolved: Okta has mitigated an issue regarding increased error rates and latency in EU1. Additional root cause information will be provided on Monday, December 14th 2020.","Status__c":"Resolved","Start_Date__c":"2020-12-09","End_Date__c":"2020-12-09","Last_Updated__c":"2020-12-15T16:25:11.000+0000","LastModifiedDate":"2020-12-15T16:25:11.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Medium","Impacted_Cells__c":"EU1","Impacted_Audience__c":"Admin;API Products;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform;MFA;Single Sign-On","Start_Time__c":"2020-12-10T06:18:00.000+0000","CreatedDate":"2020-12-10T06:20:32.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008PeuUAE"},"Id":"a9C1Y0000008PeuUAE","Duration__c":3104,"Log__c":"While iOS 12 is not officially supported, Okta has identified an issue with Okta Verify version 6.0.1 that is impacting iOS 12 users. We are working to address this issue and will provide an update at 10AM PST. There is no impact to Okta Verify on Android.\r\n\r\n====\r\n\r\nResolved: For iOS12 users experiencing crashing on Okta Verify 6.0.1, users may now downgrade to a previously functioning version by fully uninstalling the app and installing version 5.9.0. Detailed instructions for users can be found below: \r\n\r\nIf you are on iOS 12 and Okta Verify 6.0.1, and it crashes regularly, there are two options to restore service.\r\n\r\nIf you do not urgently need to use Okta Verify:\r\n 1) Wait to update to the Okta Verify 6.0.4 release.\r\n 2) This release should be ready as soon as Apple approves it (typically 1-3 business days).\r\n\r\n\r\nIf you urgently need to use Okta Verify (may require support of your admin):\r\n 1) Uninstall Okta Verify 6.0.1.\r\n 2) Install the Okta Verify app again from the App Store. This time you should be able to install the 5.9.0 version of Okta Verify.\r\n 3) If you cannot get into the Okta end-user dashboard to re-enroll Okta Verify, reach out to your administrator and ask them to reset your Okta Verify MFA.\r\n 4) Once your admin resets MFA, you will be able to re-enroll from the Okta Verify 5.9.0 you installed from the App Store.","Status__c":"Resolved","Start_Date__c":"2020-11-27","End_Date__c":"2020-11-28","Last_Updated__c":"2020-12-02T16:33:09.000+0000","LastModifiedDate":"2020-12-02T16:33:09.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;okta.com:8;okta.com:9;okta.com:10;okta.com:11;okta.com:12","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"MFA","Start_Time__c":"2020-11-27T17:09:00.000+0000","CreatedDate":"2020-11-27T17:13:16.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008PepUAE"},"Id":"a9C1Y0000008PepUAE","Duration__c":1145,"Log__c":"We are investigating delays in event processing impacting System Log, Search API, and webhooks. Workflows Events associated with the Okta connector are also delayed. We will provide an update by 12pm PST.\r\n\r\n\r\n====\r\n\r\n\r\nResolved: The processing delays have also been resolved for OP1 and System Logs are up-to-date. Additional root cause information will be provided on Monday, Nov 30th.","Status__c":"Resolved","Start_Date__c":"2020-11-25","Last_Updated__c":"2020-12-01T06:11:20.000+0000","LastModifiedDate":"2020-12-01T06:11:20.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta.com:1;okta.com:2;okta.com:3;okta.com:4;oktapreview.com:1","Impacted_Audience__c":"Admin","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2020-11-25T17:55:00.000+0000","CreatedDate":"2020-11-25T17:56:48.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008Pd8UAE"},"Id":"a9C1Y0000008Pd8UAE","Duration__c":50,"Log__c":"Okta is currently investigating an issue with Advanced Server Access. Customers attempting to authenticate using ASA may experience increased latency or failures. We will provide additional updates shortly.\r\n\r\n===\r\n\r\nService is fully restored for Advanced Server Access. Additional root cause information will be provided within 48 business hours.","Status__c":"Resolved","Start_Date__c":"2020-10-27","Last_Updated__c":"2020-10-29T21:30:38.000+0000","LastModifiedDate":"2020-10-29T21:30:38.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Medium","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Advanced Server Access","Start_Time__c":"2020-10-27T22:12:00.000+0000","CreatedDate":"2020-10-27T22:14:05.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008Pa9UAE"},"Id":"a9C1Y0000008Pa9UAE","Duration__c":866,"Log__c":"Okta is currently experiencing an issue with the service. Okta is currently investigating an issue impacting Microsoft applications in all cells where SSO & provisioning are experiencing failures. Okta is working with the vendor to restore service.\r\n\r\nRESOLVED: Issues impacting O365 Single Sign-On and 0365 Provisioning have been resolved. Okta and Microsoft have resolved the remaining intermittent provisioning issues. Okta will continue to monitor the situation carefully. For customers using AAD graph API, residual errors may be observed when Okta tries to connect to Microsoft. To resolve these issues, administrators can go to provisioning tab for the application instances, click edit, test credential and save the app instances to resolve the issue.","Status__c":"Resolved","Start_Date__c":"2020-09-28","End_Date__c":"2020-09-29","Last_Updated__c":"2020-09-30T15:50:02.000+0000","LastModifiedDate":"2020-09-30T15:50:02.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"EU1;OK1;OK2;OK3;OK4;OK5;OK6;Preview;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;OK7;Preview2;okta.com:8;okta.com:9;okta.com:10;okta.com:11","Impacted_Audience__c":"API Products;End user","Service_Feature__c":"Third Party","Okta_Sub_Service__c":"API Services","Start_Time__c":"2020-09-28T22:23:00.000+0000","CreatedDate":"2020-09-28T22:31:06.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008PZGUA2"},"Id":"a9C1Y0000008PZGUA2","Duration__c":583,"Log__c":"An issue with the Helpdesk Admin Role is preventing Admins from resetting password via temporary password. Reset password link is still functioning as a viable option. All cells except OK4 and OK6 are affected.We are actively investigating and will update this message with more information as soon as we have it.","Status__c":"Resolved","Start_Date__c":"2020-08-26","Last_Updated__c":"2020-08-28T21:46:00.000+0000","LastModifiedDate":"2020-08-28T21:46:00.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"Admin UI","Impacted_Cells__c":"EU1;OK1;OK2;OK3;OK5;Preview;OK7;Preview2;okta.com:8;okta.com:9;okta.com:10;okta.com:11","Impacted_Audience__c":"Admin","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2020-08-26T16:26:00.000+0000","CreatedDate":"2020-08-26T16:27:42.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000008PYSUA2"},"Id":"a9C1Y0000008PYSUA2","Duration__c":59,"Log__c":"Okta is currently experiencing an issue with the service. Okta is currently investigating an issue impacting US Cells 1 - 4. More detail to be provided shortly. We are actively investigating and will update this message with more information as soon as we have it.\r\n\r\nResolved: Okta has successfully restored all services in US Cell 1 - 4. Okta will continue to monitor the situation carefully. Additional root cause information will be provided within 48 hours.","Status__c":"Resolved","Start_Date__c":"2020-08-20","Last_Updated__c":"2020-08-27T17:02:59.000+0000","LastModifiedDate":"2020-08-27T17:02:59.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta.com:1;okta.com:2;okta.com:3;okta.com:4","Impacted_Audience__c":"Admin;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform;API Services;MFA;Single Sign-On","Start_Time__c":"2020-08-20T22:43:00.000+0000","CreatedDate":"2020-08-20T22:47:09.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004EW8UAM"},"Id":"a9C1Y0000004EW8UAM","Duration__c":8,"Log__c":"Okta experienced an issue whereby end users, admins, and systems making API calls experienced 502 errors when attempting to connect to ORGs located in Oktapreview:1. User, API, and Admin traffic functionality has been restored and it is suggested that those with systems making API calls verify those systems have reconnected successfully.","Status__c":"Resolved","Start_Date__c":"2020-08-13","End_Date__c":"2020-08-13","Last_Updated__c":"2020-08-19T21:20:59.000+0000","LastModifiedDate":"2020-08-19T21:20:59.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Small","Incident_Title__c":"502 Response received when attempting to access an oktapreview.com ORGs in OP1","Impacted_Cells__c":"oktapreview.com:1","Impacted_Audience__c":"Admin;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2020-08-13T20:02:00.000+0000","CreatedDate":"2020-08-13T20:35:53.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004EI1UAM"},"Id":"a9C1Y0000004EI1UAM","Duration__c":2540,"Log__c":"Okta observed an issue with the new 5.0.2 version of Okta Verify for iOS. Some users were unable to verify their identity by acknowledging the push notification sent to their iPhone. They can work around this issue by entering the verification code displayed on Okta Verify.\r\n\r\nResolved: Okta has released a fix to Okta Verify (OV version 5.0.5) for users who updated to 5.0.2 and are experiencing push notifications issues.\r\n\r\nRoot Cause: On April 15th at approximately 2pm PDT, Okta began receiving reports that some users could not complete push-based Okta Verify MFA flows on iOS devices.\r\n\r\nOkta determined that this issue was unique to version 5.0.2 of Okta Verify for iOS which had begun a phased rollout earlier that same day. At 2:20pm PDT Okta suspended the rollout of Okta Verify 5.0.2 for iOS and continued to investigate the problem. Customers were advised to work around the issue by entering the verification code displayed on Okta Verify.\r\n\r\nAfter further investigation, Okta determined that push verify on Okta Verify 5.0.2 was unable to complete verification for users who had last enrolled with Okta Verify on their device prior to Okta Verify version 4.4. This was due to an incorrect handling of legacy encryption keys that were used in previous Okta Verify versions.\r\n\r\nBecause of the complexity of the issue, Okta expedited a temporary build that would rollback the functionality for users affected as the full fix was worked on to support all upgrade paths. The build was approved and published at 11:00am PST on April 16th.\r\n\r\nOkta worked on a comprehensive fix, leveraging heuristics to determine the encryption method of the stored keys in order to resolve the issue for all users. This fix was submitted to the app store and expedited as Okta Verify 5.0.7 at 5:20pm PDT on April 16th, the build was approved and published at 9:20am PDT on April 17th.\r\n\r\nOkta is expanding migration testing scenarios across more versions to prevent future occurrences as well as incorporating additional monitoring to detect similar issues more quickly in the future.","Status__c":"Resolved","Start_Date__c":"2020-04-15","End_Date__c":"2020-04-16","Last_Updated__c":"2020-04-22T00:14:46.000+0000","LastModifiedDate":"2020-04-22T00:14:46.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"EU1;OK1;OK2;OK3;OK4;OK5;OK6;Preview;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;OK7;Preview2;okta.com:8;okta.com:9;okta.com:10;okta.com:11","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"MFA","Start_Time__c":"2020-04-15T22:26:00.000+0000","CreatedDate":"2020-04-15T22:31:26.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004E65UAE"},"Id":"a9C1Y0000004E65UAE","Duration__c":184,"Log__c":"Okta is currently experiencing an issue with the service in US Cell 3. User Imports from Workday are resulting in users in No Password state. We are actively investigating and will update this message with more information as soon as we have it.\r\n\r\nResolved: Okta has resolved the issue where Workday imports for some Orgs changed users to a No Password state resulting in those users being unable to authenticate. Root Cause Analysis will be provided here within 2 business days.\r\n\r\nComplete Root Cause Analysis may be found [Here.](https://status.okta.com/#incident/a9C1Y0000004E65UAE)","Status__c":"Resolved","Start_Date__c":"2019-12-04","End_Date__c":"2019-12-04","Last_Updated__c":"2020-03-13T02:09:53.000+0000","LastModifiedDate":"2020-03-13T02:09:53.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta.com:3","Impacted_Audience__c":"Admin;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2019-12-04T21:06:00.000+0000","CreatedDate":"2019-12-04T21:07:44.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004EBtUAM"},"Id":"a9C1Y0000004EBtUAM","Duration__c":2206,"Log__c":"Okta is currently investigating an issue preventing some customers from proceeding past the API enablement step when configuring AWS in Okta. We are actively investigating and will update this message with more information as soon as we have it.\r\n\r\nRoot Cause Analysis: On February 27th at approximately 8:15 am PST, Okta received a customer report of an error in US Cell 1 when attempting to add an additional AWS account. Upon investigation, Okta found that customers in all cells would experience errors if attempting to add child AWS accounts or modify existing AWS integration configurations.\r\n\r\nIn order to prevent further issues to customer configurations, Okta immediately made all AWS app instance configurations read-only. This change was rolled out to all cells by 1:41 pm PST until the root cause of the authentication issues could be identified and resolved.\r\nFurther investigation determined that the errors were the result of a recent upgrade to the AWS SDK which conflicted with Okta's existing logic for resolving STS endpoints. In the majority of configurations, Okta's logic would previously default to the Global STS endpoint, however after the upgrade the same resolution process began always returning the lexicographically first endpoint, which was the STS endpoint for the Asia Pacific region (Hong Kong). Because the Version 1 STS tokens that Okta generates are only valid in regions which are enabled by default, and Asia Pacific is not enabled by default, Okta's subsequent AssumeRole calls to that region were failing.\r\n\r\nThough the SDK upgrade was tested in some environments, the affected configuration workflow had not been incorporated into any automated or manual end-to-end testing flows; this service interruption highlighted a gap in our regression test plan within the AWS application integration.\r\nOnce Okta determined the cause of the errors, a fix was implemented to route all STS calls to the global endpoint. This fix was deployed to US Cell 1 at 8:49 pm PST and then to all cells. By February 28 at 9:01 pm PST, the update was completed and service was fully restored.\r\n\r\nOkta is currently working to update how endpoints and regions are passed to the AWS SDK to increase efficiency and reduce latency. As part of our on-going focus on reliability, Okta is also actively working to identify and audit test cases where application integrations and core infrastructure rely on common elements to prevent issues like this from occurring in the future.","Status__c":"Resolved","Start_Date__c":"2020-02-27","End_Date__c":"2020-02-27","Last_Updated__c":"2020-03-03T06:07:51.000+0000","LastModifiedDate":"2020-03-03T06:07:51.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"Inability to proceed past API enablement step when configuring AWS in Okta.","Impacted_Cells__c":"EU1;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;okta.com:8;okta.com:9;okta.com:10;okta.com:11","Impacted_Audience__c":"Admin","Service_Feature__c":"Core Service","Start_Time__c":"2020-02-27T19:07:00.000+0000","CreatedDate":"2020-02-27T19:19:32.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004EBeUAM"},"Id":"a9C1Y0000004EBeUAM","Duration__c":130,"Log__c":"Okta is currently experiencing an issue with the service in Okta Preview Cell 1, US Cell 6 and US Cell 7. The user import job is delayed. We are actively investigating and will update this message with more information as soon as we have it.\r\n\r\nResolved: Okta has addressed the delayed user import jobs issue on Okta Preview 1, US Cell 6, and US Cell 7. We will continue to monitor and additional root cause information will be provided within 48 hours.\r\n\r\nRoot Cause Analysis:On February 25th at approximately ‪1:05pm PST, Okta’s monitoring provided alerts that some import user jobs on Okta Preview 1, US Cell 6, and US Cell 7 were experiencing longer than normal wait times.\r\nUpon investigation, Okta determined that a new feature, “Schedule User Imports”, which allows imports to be scheduled for specific times was enabled for customers as part of the 2020.02.0 Monthly Production release*.* As part of release, existing scheduled interval imports were ported to the new day and time based feature which inadvertently caused a significant number of jobs to be run at a higher frequency. The increased volume of jobs resulted in increased wait times on some servers.\r\n\r\nAt approximately ‪2:45pm PST, Okta began to roll back the feature and terminate non-admin initiated jobs that were scheduled to execute on the impacted job servers. Jobs that were terminated would get triggered again on the next scheduled import. By ‪3:15pm PST, normal service was restored.\r\n\r\nOkta is re-evaluating design approaches to provide enhanced import scheduling capability, while ensuring reliability and performance at scale. Okta is also investigating enhancements to the alerting mechanism to improve diagnosis for this type of alert.","Status__c":"Resolved","Start_Date__c":"2020-02-25","End_Date__c":"2020-02-25","Last_Updated__c":"2020-02-28T10:33:03.000+0000","LastModifiedDate":"2020-02-28T10:33:03.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"OK6;oktapreview.com:1;OK7","Impacted_Audience__c":"Admin","Service_Feature__c":"Core Service","Start_Time__c":"2020-02-25T21:25:00.000+0000","CreatedDate":"2020-02-25T21:29:54.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004E8uUAE"},"Id":"a9C1Y0000004E8uUAE","Duration__c":321,"Log__c":"Okta is currently experiencing an issue with the service in US Cell 1. AD Agent and IWA agent states may not be correctly reflected. We are actively investigating and will update this message with more information as soon as we have it.\r\n\r\nResolved: Okta has resolved an issue in US Cell 1 impacting AD/LDAP Agent state reporting. All on-line AD/LDAP Agents are now correctly reporting ACTIVE status. Additionally, some customers where all AD/LDAP Agents were marked NOT_CONNECTED should now see successful imports. Additional root cause information will be provided within 48 hours.\r\n\r\nRoot Cause Analysis: On January 22nd at approximately ‪11:39am PST‬ Okta’s monitoring systems detected internal server errors in US Cell 1. The errors resulted in some customers' AD and LDAP agents being reported as NOT_CONNECTED to Okta Admins, even though the agents were functioning normally. Additionally, customers where all AD Agents were marked offline and who had Desktop Single Sign-On (IWA) configured using automatic fail-over may have experienced IWA failures. Okta implemented mitigation steps and the errors subsided at approximately ‪12:06pm PST‬. Okta subsequently corrected agent status for all customers by approximately ‪5:00pm PST‬.\r\n\r\nUpon investigation, Okta determined that a mistake was made when updating references to the database system during the execution of a planned routine database maintenance, This misconfiguration caused a very small number of requests to return an HTTP 500 error code and also resulted in the outdated agent status information.\r\n\r\nOkta has updated run-books and is improving the automation of maintenance verification procedures to prevent future occurrences. Okta is also investigating detection of agent status check job failures to prevent incorrect status indication to administrators.","Status__c":"Resolved","Start_Date__c":"2020-01-22","End_Date__c":"2020-01-22","Last_Updated__c":"2020-01-27T20:18:33.000+0000","LastModifiedDate":"2020-01-27T20:18:33.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta.com:1","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Start_Time__c":"2020-01-22T23:33:00.000+0000","CreatedDate":"2020-01-22T23:35:32.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004E7hUAE"},"Id":"a9C1Y0000004E7hUAE","Duration__c":59,"Log__c":"Okta had detected an issue causing increased error rates and response times for requests to OK4. The incident has been resolved. We implemented mitigating steps to return the service to normal as quickly as possible.\r\n\r\nResolved: An issue causing increased errors and response times for end-users and administrators in OK4 was resolved. During this event, End-Users and Admins may have encountered errors. Additional root cause information will provided within 48 hours.\r\n\r\nRoot Cause Analysis: Full RCA details for this event can be found [here](https://status.okta.com/#incident/a9C1Y0000004E7hUAE).","Status__c":"Resolved","Start_Date__c":"2020-01-05","End_Date__c":"2020-01-05","Last_Updated__c":"2020-01-08T14:21:11.000+0000","LastModifiedDate":"2020-01-08T14:21:11.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"OK4","Service_Feature__c":"Core Service","Start_Time__c":"2020-01-05T10:16:00.000+0000","CreatedDate":"2020-01-05T10:30:02.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004E73UAE"},"Id":"a9C1Y0000004E73UAE","Duration__c":107,"Log__c":"Okta is currently experiencing an issue with the service. We are actively investigating and will update this message with more information as soon as we have it.\r\n\r\nResolved: Okta has resolved the issue. All systems are operational issue has been mitigated. Okta will provide an RCA within 2 Business Days.\r\n\r\nRoot Cause Analysis: On December 16, 2019 at approximately 6:22 AM PST, Okta detected a degradation in application performance in US Cell 4. As a result, customers may have experienced:\r\n- Longer than normal response times\r\n- A small percentage (0.25%) experienced sporadic read-only errors\r\n- Admin users may have intermittently seen the read-only banner\r\n\r\nUpon investigation, Okta determined that the root cause was unusual database activity required to process a significant number of group rules and assignments. Although our throttling and rate limiting mechanisms engaged, a portion of requests saw additional latency due to the increased utilization of our database tier while the load was being redistributed.\r\nAt approximately 6:48 AM PST, Okta began to redistribute requests across the cell’s database cluster to reduce load and contention on the impacted database. By 8:09 AM PST, normal service performance was fully restored.\r\n\r\nOkta is implementing additional monitoring, optimization, and mitigation strategies to prevent similar issues in the future. Okta is also investigating asynchronous processing for group rules and performance improvements for group deletions to prevent system slowdown.","Status__c":"Resolved","Start_Date__c":"2019-12-16","End_Date__c":"2019-12-16","Last_Updated__c":"2019-12-19T02:29:21.000+0000","LastModifiedDate":"2019-12-19T02:29:21.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"Performance Degradation on Cell OK4","Impacted_Cells__c":"okta.com:4","Impacted_Audience__c":"Admin;API Products;End user","Service_Feature__c":"Core Service","Start_Time__c":"2019-12-16T15:31:00.000+0000","CreatedDate":"2019-12-16T15:32:34.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004E6jUAE"},"Id":"a9C1Y0000004E6jUAE","Duration__c":78,"Log__c":"Okta experienced an issue with Agentless Desktop SSO for a subset of ORGs. We are actively investigating and will update this message with more information as soon as we have it.\r\n\r\nResolved: Okta has resolved an issue where a subset of customers were unable to authenticate successfully via Agentless Desktop SSO in US Cell 1, US Cell 11, and EMEA Cell 1. Impacted customers may be required to clear their DNS cache to fully restore end user normalcy. Root Cause Analysis will be provided here within 2 business days.\r\n\r\nRoot Cause Analysis: On December 10th, 2019 at approximately 9:30 AM PST, Okta began investigating reports by customers of problems authenticating via Agent-less Desktop SSO in US Cell 1, US Cell 11, and EU Cell 1. \r\n\r\nOkta determined that during the implementation of new load-balancers on December 4th 2019 at 3:15pm PST, DNS entries for Kerberos endpoints were not updated correctly for the impacted cells. As a result, Agent-less Desktop SSO traffic was still directed to the previous load balancers which were no longer working as intended, resulting in requests that timed out and user browser sessions being redirected to Okta login screens. While affected users were able to login, they had to type in their credentials instead of being logged in automatically.\r\n\r\nAt approximately 10:48 AM, Okta updated the affected entries to reference the new load-balancers and service was restored. Some clients may have had to refresh DNS caches to restore service more quickly.\r\n\r\nOkta is working to implement additional monitoring and automation of changes to DNS records to prevent similar issues during future updates.","Status__c":"Resolved","Start_Date__c":"2019-12-10","End_Date__c":"2019-12-10","Last_Updated__c":"2019-12-16T08:19:31.000+0000","LastModifiedDate":"2019-12-16T08:19:31.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta-emea.com:1;okta.com:1;okta.com:11","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Start_Time__c":"2019-12-10T18:05:00.000+0000","CreatedDate":"2019-12-10T18:07:38.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004E5WUAU"},"Id":"a9C1Y0000004E5WUAU","Duration__c":151,"Log__c":"Okta is currently experiencing an issue with job processing for a subset of customers in US Cell 7. We are actively investigating and will update this message with more information as soon as we have it.\r\n\r\nResolved: An issue impacting job processing for a subset of customers within US Cell 7 has been resolved. The remaining queued jobs are processing at full capacity and we will continue to monitor. Root Cause Analysis is underway and we will post full RCA details here within 48 hours.\r\n\r\nRoot Cause Analysis: On December 2nd at approximately 7:45AM (PST), Okta detected a slowdown with processing import jobs on US Cell 7 and subsequently determined that there was an unusual and significant increase in asynchronous job workload which resulted in a backup of job processing.\r\n\r\nOkta increased system resources to expedite the processing of pending jobs. This mitigated the issue and normal functionality was fully restored by ‪10:16 AM.\r\n\r\nOkta is working optimize the current implementation of group application assignments to batch and de-duplicate assignments. Okta is also adding monitoring and controls to detect and mitigate similar issues in the future","Status__c":"Resolved","Start_Date__c":"2019-12-02","End_Date__c":"2019-12-02","Last_Updated__c":"2019-12-08T03:03:48.000+0000","LastModifiedDate":"2019-12-08T03:03:48.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta.com:7","Impacted_Audience__c":"Admin","Service_Feature__c":"Core Service","Start_Time__c":"2019-12-02T16:34:00.000+0000","CreatedDate":"2019-12-02T16:34:59.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004E5RUAU"},"Id":"a9C1Y0000004E5RUAU","Duration__c":342,"Log__c":"Okta is currently experiencing an issue with Advanced Server Access. Administrators may have trouble accessing the dashboard, and users may experience errors with SSO or while attempting to authenticate to servers. We are actively investigating and will update this message with more information as soon as we have it.\r\n\r\nResolved: An issue impacting Okta Advanced Server Access has been resolved. During the event, Administrators may have experienced errors accessing the ASA dashboard. Additionally, users may have experienced authentication errors. Okta applied several changes to mitigate the issue and continues to monitor and investigate. Additional root cause information will be provided within 48 business hours.\r\n\r\nRoot Cause Analysis: On Saturday, November 23rd at approximately 1:05 PM PDT, Okta detected database resource contention within our Advanced Server Access production environment, resulting in timeout errors for users who were accessing the Advanced Server Access dashboard or were attempting to authenticate to servers.\r\n\r\nAfter investigating the issue, at approximately 3:30 PM PDT, Okta took actions to mitigate the resource contention issue by blocking customer API requests for adding groups to projects and synchronizing data to servers. Most Advanced Server Access functionality was restored through this action. Upon further investigation, Okta identified an additional inefficiency in our synchronous logic, and made a change to asynchronous background job processing.\r\n\r\nFollowing our investigation and mitigation actions, Okta re-enabled the API for synchronizing data to servers at approximately 6:47 PM PDT. Okta also re-enabled the API for adding groups to projects, fully restoring all services at this time.\r\n\r\nOkta's has implemented changes to mitigate the impact of similar issues in the future, and is continuing its full Root Cause Analysis of the incident,. We expect to be able to provide further details and future preventative actions by December 4, 2019. Additional RCA details will be posted here once this is investigation is complete.","Status__c":"Resolved","Start_Date__c":"2019-11-23","End_Date__c":"2019-11-23","Last_Updated__c":"2019-12-03T21:49:47.000+0000","LastModifiedDate":"2019-12-03T21:49:47.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"EU1;OK1;OK2;OK3;OK4;OK5;OK6;Preview;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;OK7;Preview2;okta.com:8;okta.com:9;okta.com:10;okta.com:11","Impacted_Audience__c":"Admin;End user","Service_Feature__c":"Core Service","Start_Time__c":"2019-11-23T23:06:00.000+0000","CreatedDate":"2019-11-23T23:07:45.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004E4EUAU"},"Id":"a9C1Y0000004E4EUAU","Duration__c":684,"Log__c":"Okta customers in all cells that leverage Zoom’s integration for provisioning are experiencing errors while updating users in Zoom. We are actively working with Zoom to identify the root cause and identify a solution.\r\n\r\nResolved: An issue impacting user provisioning updates to the Zoom application within Okta has been resolved. During the event, user updates to the Zoom application may have encountered errors if the provisioned user already existed within Zoom. Okta worked with Zoom to identify the root cause of the issue and applied a hot-fix to all production cells to mitigate the provisioning errors while Zoom worked to deploy a permanent fix. Additional root cause information will be provided within 48 business hours.\r\n\r\nRoot Cause Analysis: On Nov 19th at approximately 8:53AM PDT, customers reported Zoom provisioning errors when updating user’s profiles.\r\n\r\nUpon investigation, Okta determined that the issue was caused by a change in the behavior of an existing Zoom Provisioning API which was introduced on November 17. This change was implemented by Zoom to enable support for changing user's email addresses through the Profile Update request. The change inadvertently caused profile updates to fail, whenever the email has not been changed.\r\n\r\nOn Nov 19th at approximately 8:17 PM PDT Okta released a work around to address the issue. This update was deployed to all production cells and service was fully restored.\r\n\r\nOkta is working closely with Zoom for a fix, as well as, understanding and mitigating the re-occurrence for this type of failure","Status__c":"Resolved","Start_Date__c":"2019-11-19","End_Date__c":"2019-11-19","Last_Updated__c":"2019-11-22T17:31:40.000+0000","LastModifiedDate":"2019-11-22T17:31:40.000+0000","Category__c":"Feature Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"EU1;OK1;OK2;OK3;OK4;OK5;OK6;Preview;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;OK7;Preview2;okta.com:8;okta.com:9;okta.com:10;okta.com:11","Impacted_Audience__c":"Admin;End user","Service_Feature__c":"Third Party","Start_Time__c":"2019-11-19T17:44:00.000+0000","CreatedDate":"2019-11-19T17:47:37.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004E4JUAU"},"Id":"a9C1Y0000004E4JUAU","Duration__c":495,"Log__c":"Okta is investigating an issue with installations of Okta Mobile on Android. We will provide more information once available.\r\n\r\nResolved: Okta has addressed an issue on Okta Mobile for Android for the vast majority of Okta end users. When opening the Okta Mobile for Android application, end users may be logged out of their account and encounter the following message: “Detected possible malicious activity when establishing a secure connection to Okta”. Okta Mobile Android users on Android 6.0 or higher must update the Okta Mobile application to the latest version through the Google Play Store. Okta continues to investigate a solution for users running versions of Android below 6.0 as well as those relying on downloading the Okta Mobile APK from their Okta tenant. We apologize for the inconvenience. Additional root cause information will be provided within 48 business hours.\r\n\r\nRoot Cause Analysis: On Nov 19th at approximately 12:30 PM PDT, customers reported that when opening the Okta Mobile application for Android, end users were intermittently logged out of their account and encountered the following message: \r\n\r\n “Detected possible malicious activity when establishing a secure connection to Okta”.\r\n\r\nUpon investigation, Okta determined that the issue was caused by an update to the preferred authentication algorithm used in TLS by Okta Mobile’s error tracking provider. The new preferred authentication algorithm chosen by the provider became ECDSA, which was not allowed by the Okta Mobile Android application. Upon detecting a change in the encryption algorithm in use, Okta Mobile on Android is designed to present the aforementioned warning message.\r\n\r\nOn Nov 19th at approximately 8:45 PM PDT, Okta released Android Okta Mobile version 4.3.2, for Android versions 6.0 and above, and notified Org administrators through the Admin Console banner that users should update their Okta Mobile application to address the issue. Users running an older version of Android are encouraged to update their device to 6.0 or later and install the latest version of Okta Mobile.\r\n\r\nOkta is implementing additional monitoring and alerting to detect such issues earlier in the future. Additionally, the latest Android Okta Mobile version has expanded the set of authentication algorithms allowed and newer versions will include an improved design to prevent issues like this in the future.","Status__c":"Resolved","Start_Date__c":"2019-11-19","End_Date__c":"2019-11-19","Last_Updated__c":"2019-11-22T16:13:38.000+0000","LastModifiedDate":"2019-11-22T16:13:38.000+0000","Category__c":"Service Degradation","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"EU1;OK1;OK2;OK3;OK4;OK5;OK6;Preview;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;OK7;Preview2;okta.com:8;okta.com:9;okta.com:10;okta.com:11","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Start_Time__c":"2019-11-19T20:49:00.000+0000","CreatedDate":"2019-11-19T20:54:32.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v53.0/sobjects/Incident__c/a9C1Y0000004E2wUAE"},"Id":"a9C1Y0000004E2wUAE","Duration__c":125,"Log__c":"Okta is currently experiencing an issue with the service. We are actively investigating and will update this message with more information as soon as we have it. Office 365 policies are denying access to users.\r\n\r\nResolved: \r\n\r\nOkta has successfully deployed and verified a fix across all Cells. The identified issue impacted a subset of the users in all cells using legacy protocol sign-on with Office 365 or service accounts. Additional root cause information will provided within 48 hours.\r\n\r\nRoot Cause Analysis: \r\n\r\nOn Nov 6th at approximately 9:30 AM PDT, Okta received reports from certain customers across multiple cells that some users were not able to log into the Okta Service via Office 365 clients using legacy protocols such as the Outlook Desktop App, and some service accounts used for automation.\r\n\r\nOnly end users using Outlook clients with an Exchange ActiveSync profile and without an enrolled second factor that are members of tenants with a configured MFA Enrollment Policy were impacted.\r\n\r\nThe pursuant investigation determined that a bug fix, applied in the 2019.10.2 weekly update, to address a gap in multi-factor enrollment during authentication, was the root cause. The fix was incorrectly scoped to include legacy Office 365 protocols.\r\n\r\nAt approximately 10:44 AM PDT, Okta reverted the change in US Cell 4 and confirmed that issue was resolved. The change was then subsequently reverted in all production. At approximately 11:35 AM, service was fully restored for all customers in all cells.\r\n\r\nAdditional testing and alerting to prevent and detect such issues will be implemented to prevent future occurrences.","Status__c":"Resolved","Start_Date__c":"2019-11-06","End_Date__c":"2019-11-06","Last_Updated__c":"2019-11-10T17:25:07.000+0000","LastModifiedDate":"2019-11-10T17:25:07.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Incident_Title__c":"Office 365 policies are deny access to users.","Impacted_Cells__c":"EU1;OK1;OK2;OK3;OK4;OK5;OK6;Preview;okta-emea.com:1;okta.com:1;okta.com:2;okta.com:3;okta.com:4;okta.com:5;okta.com:6;okta.com:7;oktapreview.com:1;oktapreview.com:2;OK7;Preview2;okta.com:8;okta.com:9;okta.com:10;okta.com:11","Impacted_Audience__c":"End user","Service_Feature__c":"Core Service","Start_Time__c":"2019-11-06T17:32:00.000+0000","CreatedDate":"2019-11-06T17:35:16.000+0000","CurrencyIsoCode":"USD"}][]Okta Status Site Guest UserGuestokta.com:1,okta.com:2,okta.com:3,okta.com:4,okta.com:5,okta.com:6,okta.com:7,okta.com:8,okta.com:9,okta.com:10,okta.com:11,okta.com:12,okta.com:14,okta.com:15,okta-gov.com:1,okta-emea.com:1,oktapreview.com:1,oktapreview.com:2,oktapreview.com:3[{"year":2010, "uptime":99.98, "month":["100.000","100.000","100.000","100.000","99.955","100.000","100.000","99.933","99.928","99.960","100.000","100.000"]},{"year":2011, "uptime":99.96, "month":["99.973","99.980","100.000","99.977","99.969","99.954","100.000","99.917","99.889","99.964","100.000","99.953"]},{"year":2012, "uptime":99.9966, "month":["100.000","100.000","99.996","100.000","100.000","100.000","100.000","99.964","100.000","100.000","100.000","100.000"]},{"year":2013, "uptime":99.9994, "month":["99.993","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000"]},{"year":2014, "uptime":99.95, "month":["100.000","100.000","100.000","100.000","100.000","100.000","100.000","99.610","99.824","99.993","100.000","99.980"]},{"year":2015, "uptime":100.0000, "month":["100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000"]},{"year":2016, "uptime":99.95, "month":["100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","99.384","100.000","100.000"]},{"year":2017, "uptime":99.97, "month":["99.969","99.692","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000"]},{"year":2018, "uptime":100.0000, "month":["100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000"]},{"year":2019, "uptime":100.0000, "month":["100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000"]},{"year":2020, "uptime":100.0000, "month":["100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000"]},{"year":2021, "uptime":100.0000, "month":["100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000","100.000"]}][{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00019joFtkAAE"},"Id":"0174z00019joFtkAAE","ParentId":"a9C4z000000wk5cEAA","CreatedDate":"2021-11-08T20:48:44.000+0000","Field":"Formatted_Audience__c","OldValue":null,"NewValue":"Admin"},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00019joFtlAAE"},"Id":"0174z00019joFtlAAE","ParentId":"a9C4z000000wk5cEAA","CreatedDate":"2021-11-08T20:48:44.000+0000","Field":"Formatted_Cells__c","OldValue":null,"NewValue":"OK4, US Cell 4, US Cell 7, OK7, US Cell 11"},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00019joGUXAA2"},"Id":"0174z00019joGUXAA2","ParentId":"a9C4z000000wk5cEAA","CreatedDate":"2021-11-08T20:48:44.000+0000","Field":"Formatted_Service_Features__c","OldValue":null,"NewValue":"Core Service"},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00019joEuYAAU"},"Id":"0174z00019joEuYAAU","ParentId":"a9C4z000000wk5cEAA","CreatedDate":"2021-11-08T20:50:13.000+0000","Field":"Impacted_Cells__c","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00019jobPPAAY"},"Id":"0174z00019jobPPAAY","ParentId":"a9C4z000000wk5cEAA","CreatedDate":"2021-11-08T21:56:53.000+0000","Field":"Impacted_Cells__c","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00019jodYCAAY"},"Id":"0174z00019jodYCAAY","ParentId":"a9C4z000000wk5cEAA","CreatedDate":"2021-11-08T22:03:27.000+0000","Field":"Impacted_Cells__c","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000XWliV8QQJ"},"Id":"0172A0000XWliV8QQJ","ParentId":"a9C2A000000XZVuUAO","CreatedDate":"2016-10-21T12:32:05.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00011I0ZDGQA3"},"Id":"0171Y00011I0ZDGQA3","ParentId":"a9C1Y0000004EW8UAM","CreatedDate":"2020-08-13T20:35:54.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00011itu4hQAA"},"Id":"0171Y00011itu4hQAA","ParentId":"a9C1Y0000008PZGUA2","CreatedDate":"2020-08-26T16:27:43.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000ZBIEbYQQX"},"Id":"0172A0000ZBIEbYQQX","ParentId":"a9C2A000000XZl4UAG","CreatedDate":"2017-03-21T03:00:43.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000XAa1hOQQR"},"Id":"0172A0000XAa1hOQQR","ParentId":"a9C2A000000PBZNUA4","CreatedDate":"2016-09-28T19:56:21.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000bPuQwbQAF"},"Id":"0172A0000bPuQwbQAF","ParentId":"a9C2A000000KyvbUAC","CreatedDate":"2017-09-13T14:33:13.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000lI4QbtQAF"},"Id":"0171Y0000lI4QbtQAF","ParentId":"a9C1Y000000GpKxUAK","CreatedDate":"2019-02-08T00:36:34.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000eIqo6FQAR"},"Id":"0172A0000eIqo6FQAR","ParentId":"a9C2A000000bmK1UAI","CreatedDate":"2018-03-30T14:36:53.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000bv1PXsQAM"},"Id":"0172A0000bv1PXsQAM","ParentId":"a9C2A000000Kz0CUAS","CreatedDate":"2017-10-19T17:04:56.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000LFUUgFIQX"},"Id":"017F00000LFUUgFIQX","ParentId":"a9CF0000000PBLVMA4","CreatedDate":"2015-06-30T03:16:11.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000LrqPhlIQE"},"Id":"017F00000LrqPhlIQE","ParentId":"a9CF0000000PBR4MAO","CreatedDate":"2015-07-27T12:15:12.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00011UucezQAB"},"Id":"0171Y00011UucezQAB","ParentId":"a9C1Y0000008PYSUA2","CreatedDate":"2020-08-20T22:47:09.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000bGmbCSQAZ"},"Id":"0172A0000bGmbCSQAZ","ParentId":"a9C2A000000Kyv2UAC","CreatedDate":"2017-08-31T17:37:43.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000bWVjUiQAL"},"Id":"0170Z0000bWVjUiQAL","ParentId":"a9C0Z000000TNgDUAW","CreatedDate":"2018-05-17T18:49:34.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000dq55bAQAQ"},"Id":"0170Z0000dq55bAQAQ","ParentId":"a9C0Z0000008PTrUAM","CreatedDate":"2018-09-04T13:39:52.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000rEQNXBQA5"},"Id":"0171Y0000rEQNXBQA5","ParentId":"a9C1Y0000004DyLUAU","CreatedDate":"2019-09-30T22:50:16.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000qYmFzTQAV"},"Id":"0171Y0000qYmFzTQAV","ParentId":"a9C1Y000000fxj2UAA","CreatedDate":"2019-09-05T19:31:34.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000ZmpUXIQQ2"},"Id":"0172A0000ZmpUXIQQ2","ParentId":"a9C2A000000blcGUAQ","CreatedDate":"2017-05-05T17:07:11.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YS5CkxQQF"},"Id":"0172A0000YS5CkxQQF","ParentId":"a9C2A000000XZeNUAW","CreatedDate":"2017-01-17T19:34:18.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000aDMIbhQAH"},"Id":"0172A0000aDMIbhQAH","ParentId":"a9C2A000000bm6BUAQ","CreatedDate":"2017-06-14T03:15:51.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000cYSj9RQAT"},"Id":"0170Z0000cYSj9RQAT","ParentId":"a9C0Z0000004EAAUA2","CreatedDate":"2018-07-05T18:15:09.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000b2IZ3NQAW"},"Id":"0170Z0000b2IZ3NQAW","ParentId":"a9C0Z000000TNelUAG","CreatedDate":"2018-04-23T16:18:41.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000k28E8gQAE"},"Id":"0171Y0000k28E8gQAE","ParentId":"a9C1Y000000GpJuUAK","CreatedDate":"2018-12-13T13:04:33.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JVXZjpIQH"},"Id":"017F00000JVXZjpIQH","ParentId":"a9CF0000000PBAhMAO","CreatedDate":"2015-04-17T20:01:55.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yHYIQZ"},"Id":"017F00000JQ9yHYIQZ","ParentId":"a9CF0000000PB78MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yHZIQZ"},"Id":"017F00000JQ9yHZIQZ","ParentId":"a9CF0000000PB79MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yHaIQJ"},"Id":"017F00000JQ9yHaIQJ","ParentId":"a9CF0000000PB7AMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNiIQJ"},"Id":"017F00000JQ9yNiIQJ","ParentId":"a9CF0000000PB7BMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNjIQJ"},"Id":"017F00000JQ9yNjIQJ","ParentId":"a9CF0000000PB7CMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNkIQJ"},"Id":"017F00000JQ9yNkIQJ","ParentId":"a9CF0000000PB7DMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNlIQJ"},"Id":"017F00000JQ9yNlIQJ","ParentId":"a9CF0000000PB7EMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNnIQJ"},"Id":"017F00000JQ9yNnIQJ","ParentId":"a9CF0000000PB7GMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNoIQJ"},"Id":"017F00000JQ9yNoIQJ","ParentId":"a9CF0000000PB7HMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNpIQJ"},"Id":"017F00000JQ9yNpIQJ","ParentId":"a9CF0000000PB7IMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNqIQJ"},"Id":"017F00000JQ9yNqIQJ","ParentId":"a9CF0000000PB7JMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNrIQJ"},"Id":"017F00000JQ9yNrIQJ","ParentId":"a9CF0000000PB7KMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNsIQJ"},"Id":"017F00000JQ9yNsIQJ","ParentId":"a9CF0000000PB7LMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNtIQJ"},"Id":"017F00000JQ9yNtIQJ","ParentId":"a9CF0000000PB7MMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNuIQJ"},"Id":"017F00000JQ9yNuIQJ","ParentId":"a9CF0000000PB7NMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNvIQJ"},"Id":"017F00000JQ9yNvIQJ","ParentId":"a9CF0000000PB7OMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNwIQJ"},"Id":"017F00000JQ9yNwIQJ","ParentId":"a9CF0000000PB7PMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNxIQJ"},"Id":"017F00000JQ9yNxIQJ","ParentId":"a9CF0000000PB7QMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNyIQJ"},"Id":"017F00000JQ9yNyIQJ","ParentId":"a9CF0000000PB7RMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yNzIQJ"},"Id":"017F00000JQ9yNzIQJ","ParentId":"a9CF0000000PB7SMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQA7u8IQD"},"Id":"017F00000JQA7u8IQD","ParentId":"a9CF0000000PB8gMAG","CreatedDate":"2015-04-14T05:39:20.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yO1IQJ"},"Id":"017F00000JQ9yO1IQJ","ParentId":"a9CF0000000PB7UMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yO2IQJ"},"Id":"017F00000JQ9yO2IQJ","ParentId":"a9CF0000000PB7VMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yO3IQJ"},"Id":"017F00000JQ9yO3IQJ","ParentId":"a9CF0000000PB7WMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yO4IQJ"},"Id":"017F00000JQ9yO4IQJ","ParentId":"a9CF0000000PB7XMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yO5IQJ"},"Id":"017F00000JQ9yO5IQJ","ParentId":"a9CF0000000PB7YMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yO6IQJ"},"Id":"017F00000JQ9yO6IQJ","ParentId":"a9CF0000000PB7ZMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yO7IQJ"},"Id":"017F00000JQ9yO7IQJ","ParentId":"a9CF0000000PB7aMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yO8IQJ"},"Id":"017F00000JQ9yO8IQJ","ParentId":"a9CF0000000PB7bMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yO9IQJ"},"Id":"017F00000JQ9yO9IQJ","ParentId":"a9CF0000000PB7cMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOAIQZ"},"Id":"017F00000JQ9yOAIQZ","ParentId":"a9CF0000000PB7dMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOBIQZ"},"Id":"017F00000JQ9yOBIQZ","ParentId":"a9CF0000000PB7eMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOCIQZ"},"Id":"017F00000JQ9yOCIQZ","ParentId":"a9CF0000000PB7fMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yODIQZ"},"Id":"017F00000JQ9yODIQZ","ParentId":"a9CF0000000PB7gMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOEIQZ"},"Id":"017F00000JQ9yOEIQZ","ParentId":"a9CF0000000PB7hMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOFIQZ"},"Id":"017F00000JQ9yOFIQZ","ParentId":"a9CF0000000PB7iMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOGIQZ"},"Id":"017F00000JQ9yOGIQZ","ParentId":"a9CF0000000PB7jMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOHIQZ"},"Id":"017F00000JQ9yOHIQZ","ParentId":"a9CF0000000PB7kMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOIIQZ"},"Id":"017F00000JQ9yOIIQZ","ParentId":"a9CF0000000PB7lMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOJIQZ"},"Id":"017F00000JQ9yOJIQZ","ParentId":"a9CF0000000PB7mMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOKIQZ"},"Id":"017F00000JQ9yOKIQZ","ParentId":"a9CF0000000PB7nMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOLIQZ"},"Id":"017F00000JQ9yOLIQZ","ParentId":"a9CF0000000PB7oMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOMIQZ"},"Id":"017F00000JQ9yOMIQZ","ParentId":"a9CF0000000PB7pMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yO0IQJ"},"Id":"017F00000JQ9yO0IQJ","ParentId":"a9CF0000000PB7TMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yONIQZ"},"Id":"017F00000JQ9yONIQZ","ParentId":"a9CF0000000PB7qMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOPIQZ"},"Id":"017F00000JQ9yOPIQZ","ParentId":"a9CF0000000PB7sMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOQIQZ"},"Id":"017F00000JQ9yOQIQZ","ParentId":"a9CF0000000PB7tMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yORIQZ"},"Id":"017F00000JQ9yORIQZ","ParentId":"a9CF0000000PB7uMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOSIQZ"},"Id":"017F00000JQ9yOSIQZ","ParentId":"a9CF0000000PB7vMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOTIQZ"},"Id":"017F00000JQ9yOTIQZ","ParentId":"a9CF0000000PB7wMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOUIQZ"},"Id":"017F00000JQ9yOUIQZ","ParentId":"a9CF0000000PB7xMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOVIQZ"},"Id":"017F00000JQ9yOVIQZ","ParentId":"a9CF0000000PB7yMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOWIQZ"},"Id":"017F00000JQ9yOWIQZ","ParentId":"a9CF0000000PB7zMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOXIQZ"},"Id":"017F00000JQ9yOXIQZ","ParentId":"a9CF0000000PB80MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOYIQZ"},"Id":"017F00000JQ9yOYIQZ","ParentId":"a9CF0000000PB81MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOZIQZ"},"Id":"017F00000JQ9yOZIQZ","ParentId":"a9CF0000000PB82MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOaIQJ"},"Id":"017F00000JQ9yOaIQJ","ParentId":"a9CF0000000PB83MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yObIQJ"},"Id":"017F00000JQ9yObIQJ","ParentId":"a9CF0000000PB84MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOcIQJ"},"Id":"017F00000JQ9yOcIQJ","ParentId":"a9CF0000000PB85MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOdIQJ"},"Id":"017F00000JQ9yOdIQJ","ParentId":"a9CF0000000PB86MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOeIQJ"},"Id":"017F00000JQ9yOeIQJ","ParentId":"a9CF0000000PB87MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOfIQJ"},"Id":"017F00000JQ9yOfIQJ","ParentId":"a9CF0000000PB88MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOgIQJ"},"Id":"017F00000JQ9yOgIQJ","ParentId":"a9CF0000000PB89MAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOhIQJ"},"Id":"017F00000JQ9yOhIQJ","ParentId":"a9CF0000000PB8AMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOiIQJ"},"Id":"017F00000JQ9yOiIQJ","ParentId":"a9CF0000000PB8BMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOjIQJ"},"Id":"017F00000JQ9yOjIQJ","ParentId":"a9CF0000000PB8CMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOkIQJ"},"Id":"017F00000JQ9yOkIQJ","ParentId":"a9CF0000000PB8DMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOlIQJ"},"Id":"017F00000JQ9yOlIQJ","ParentId":"a9CF0000000PB8EMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOmIQJ"},"Id":"017F00000JQ9yOmIQJ","ParentId":"a9CF0000000PB8FMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOOIQZ"},"Id":"017F00000JQ9yOOIQZ","ParentId":"a9CF0000000PB7rMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOnIQJ"},"Id":"017F00000JQ9yOnIQJ","ParentId":"a9CF0000000PB8GMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOoIQJ"},"Id":"017F00000JQ9yOoIQJ","ParentId":"a9CF0000000PB8HMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOpIQJ"},"Id":"017F00000JQ9yOpIQJ","ParentId":"a9CF0000000PB8IMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOqIQJ"},"Id":"017F00000JQ9yOqIQJ","ParentId":"a9CF0000000PB8JMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOrIQJ"},"Id":"017F00000JQ9yOrIQJ","ParentId":"a9CF0000000PB8KMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOsIQJ"},"Id":"017F00000JQ9yOsIQJ","ParentId":"a9CF0000000PB8LMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOtIQJ"},"Id":"017F00000JQ9yOtIQJ","ParentId":"a9CF0000000PB8MMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOuIQJ"},"Id":"017F00000JQ9yOuIQJ","ParentId":"a9CF0000000PB8NMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOvIQJ"},"Id":"017F00000JQ9yOvIQJ","ParentId":"a9CF0000000PB8OMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOwIQJ"},"Id":"017F00000JQ9yOwIQJ","ParentId":"a9CF0000000PB8PMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOxIQJ"},"Id":"017F00000JQ9yOxIQJ","ParentId":"a9CF0000000PB8QMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOyIQJ"},"Id":"017F00000JQ9yOyIQJ","ParentId":"a9CF0000000PB8RMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yOzIQJ"},"Id":"017F00000JQ9yOzIQJ","ParentId":"a9CF0000000PB8SMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yP0IQJ"},"Id":"017F00000JQ9yP0IQJ","ParentId":"a9CF0000000PB8TMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yP1IQJ"},"Id":"017F00000JQ9yP1IQJ","ParentId":"a9CF0000000PB8UMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yP2IQJ"},"Id":"017F00000JQ9yP2IQJ","ParentId":"a9CF0000000PB8VMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yP3IQJ"},"Id":"017F00000JQ9yP3IQJ","ParentId":"a9CF0000000PB8WMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yP4IQJ"},"Id":"017F00000JQ9yP4IQJ","ParentId":"a9CF0000000PB8XMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yP5IQJ"},"Id":"017F00000JQ9yP5IQJ","ParentId":"a9CF0000000PB8YMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yP6IQJ"},"Id":"017F00000JQ9yP6IQJ","ParentId":"a9CF0000000PB8ZMAW","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yP7IQJ"},"Id":"017F00000JQ9yP7IQJ","ParentId":"a9CF0000000PB8aMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yP8IQJ"},"Id":"017F00000JQ9yP8IQJ","ParentId":"a9CF0000000PB8bMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yP9IQJ"},"Id":"017F00000JQ9yP9IQJ","ParentId":"a9CF0000000PB8cMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yPAIQZ"},"Id":"017F00000JQ9yPAIQZ","ParentId":"a9CF0000000PB8dMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yPBIQZ"},"Id":"017F00000JQ9yPBIQZ","ParentId":"a9CF0000000PB8eMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JQ9yPCIQZ"},"Id":"017F00000JQ9yPCIQZ","ParentId":"a9CF0000000PB8fMAG","CreatedDate":"2015-04-14T05:29:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000K5dqXMIQY"},"Id":"017F00000K5dqXMIQY","ParentId":"a9CF0000000PBFNMA4","CreatedDate":"2015-05-12T11:55:12.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YktEugQQE"},"Id":"0172A0000YktEugQQE","ParentId":"a9C2A000000XZiUUAW","CreatedDate":"2017-02-15T16:29:10.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YBh4fAQQR"},"Id":"0172A0000YBh4fAQQR","ParentId":"a9C2A000000XZcgUAG","CreatedDate":"2016-12-21T17:20:30.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00016xaFwEQAU"},"Id":"0171Y00016xaFwEQAU","ParentId":"a9C1Y0000008Ph5UAE","CreatedDate":"2021-02-16T20:00:22.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000bZO71eQAD"},"Id":"0172A0000bZO71eQAD","ParentId":"a9C2A000000KyxrUAC","CreatedDate":"2017-09-26T13:16:54.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YcwVFkQQM"},"Id":"0172A0000YcwVFkQQM","ParentId":"a9C2A000000XZfGUAW","CreatedDate":"2017-02-02T12:57:47.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YbiwOrQQI"},"Id":"0172A0000YbiwOrQQI","ParentId":"a9C2A000000XZewUAG","CreatedDate":"2017-01-31T21:03:34.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000sc0832QAA"},"Id":"0171Y0000sc0832QAA","ParentId":"a9C1Y0000004E4EUAU","CreatedDate":"2019-11-19T17:47:37.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000sxU133QAC"},"Id":"0171Y0000sxU133QAC","ParentId":"a9C1Y0000004E5WUAU","CreatedDate":"2019-12-02T16:34:59.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000tEZI96QAH"},"Id":"0171Y0000tEZI96QAH","ParentId":"a9C1Y0000004E6jUAE","CreatedDate":"2019-12-10T18:07:38.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000cRTRSJQA5"},"Id":"0170Z0000cRTRSJQA5","ParentId":"a9C0Z0000004EA5UAM","CreatedDate":"2018-06-29T17:40:00.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000ZY6QjkQQF"},"Id":"0172A0000ZY6QjkQQF","ParentId":"a9C2A000000XZsoUAG","CreatedDate":"2017-04-17T18:16:15.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00014RiYN5QAN"},"Id":"0171Y00014RiYN5QAN","ParentId":"a9C1Y0000008PeuUAE","CreatedDate":"2020-11-27T17:13:16.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000axTPnpQAG"},"Id":"0172A0000axTPnpQAG","ParentId":"a9C2A000000bmDeUAI","CreatedDate":"2017-08-03T12:31:20.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000bH0XiBQAV"},"Id":"0170Z0000bH0XiBQAV","ParentId":"a9C0Z000000TNfAUAW","CreatedDate":"2018-05-07T02:20:37.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000NoEk4xIQC"},"Id":"017F00000NoEk4xIQC","ParentId":"a9CF0000000blLjMAI","CreatedDate":"2015-10-15T01:12:12.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000dkdFSHQA2"},"Id":"0172A0000dkdFSHQA2","ParentId":"a9C2A000000PBiBUAW","CreatedDate":"2018-02-25T17:30:37.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000e3AmZOQA0"},"Id":"0172A0000e3AmZOQA0","ParentId":"a9C2A0000004CSTUA2","CreatedDate":"2018-03-16T17:12:34.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000b7Rg1CQAS"},"Id":"0172A0000b7Rg1CQAS","ParentId":"a9C2A000000bmHvUAI","CreatedDate":"2017-08-17T23:04:30.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YMaIhNQQV"},"Id":"0172A0000YMaIhNQQV","ParentId":"a9C2A000000XZdeUAG","CreatedDate":"2017-01-09T16:12:20.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YMaKHdQQN"},"Id":"0172A0000YMaKHdQQN","ParentId":"a9C2A000000XZdjUAG","CreatedDate":"2017-01-09T16:18:13.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000XuGYxRQQW"},"Id":"0172A0000XuGYxRQQW","ParentId":"a9C2A000000XZYKUA4","CreatedDate":"2016-11-30T01:02:28.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000sCTiwVQAT"},"Id":"0171Y0000sCTiwVQAT","ParentId":"a9C1Y0000004E2wUAE","CreatedDate":"2019-11-06T17:35:16.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000sc2DUJQA2"},"Id":"0171Y0000sc2DUJQA2","ParentId":"a9C1Y0000004E4JUAU","CreatedDate":"2019-11-19T20:54:33.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000uBnpB2QAJ"},"Id":"0171Y0000uBnpB2QAJ","ParentId":"a9C1Y0000004E7hUAE","CreatedDate":"2020-01-05T10:30:02.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000bpMJXqQAO"},"Id":"0170Z0000bpMJXqQAO","ParentId":"a9C0Z0000004E5UUAU","CreatedDate":"2018-05-31T07:35:02.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z000126tZm4AAE"},"Id":"0174z000126tZm4AAE","ParentId":"a9C4z000000oLkyEAE","CreatedDate":"2021-04-01T22:17:08.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00014O4YJuAAN"},"Id":"0174z00014O4YJuAAN","ParentId":"a9C4z000000oLmzEAE","CreatedDate":"2021-06-11T18:02:40.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000b96tesQAA"},"Id":"0172A0000b96tesQAA","ParentId":"a9C2A000000bmJDUAY","CreatedDate":"2017-08-21T17:36:30.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000lFFgbLQAT"},"Id":"0171Y0000lFFgbLQAT","ParentId":"a9C1Y000000GpKsUAK","CreatedDate":"2019-02-06T00:39:40.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000aHNE4gQAH"},"Id":"0172A0000aHNE4gQAH","ParentId":"a9C2A000000bm9IUAQ","CreatedDate":"2017-06-20T04:06:05.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000NTxzuSIQR"},"Id":"017F00000NTxzuSIQR","ParentId":"a9CF0000000XZKbMAO","CreatedDate":"2015-10-01T12:19:58.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000dtveDyQAI"},"Id":"0172A0000dtveDyQAI","ParentId":"a9C2A000000PBikUAG","CreatedDate":"2018-03-07T18:00:08.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000dQcNrEQAV"},"Id":"0172A0000dQcNrEQAV","ParentId":"a9C2A000000KzYJUA0","CreatedDate":"2018-02-02T17:48:52.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000LIDzaKIQT"},"Id":"017F00000LIDzaKIQT","ParentId":"a9CF0000000PBLaMAO","CreatedDate":"2015-07-01T19:14:26.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000ZtrmGjQQI"},"Id":"0172A0000ZtrmGjQQI","ParentId":"a9C2A000000blruUAA","CreatedDate":"2017-05-15T13:19:44.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00016ZpvQxQAJ"},"Id":"0171Y00016ZpvQxQAJ","ParentId":"a9C1Y0000008PggUAE","CreatedDate":"2021-02-03T22:09:19.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000XuJp7uQQC"},"Id":"0172A0000XuJp7uQQC","ParentId":"a9C2A000000XZYPUA4","CreatedDate":"2016-11-30T19:10:08.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00014pU866QAC"},"Id":"0171Y00014pU866QAC","ParentId":"a9C1Y0000008PfEUAU","CreatedDate":"2020-12-10T06:20:32.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000XmAKzzQQG"},"Id":"0172A0000XmAKzzQQG","ParentId":"a9C2A000000XZXbUAO","CreatedDate":"2016-11-15T20:58:57.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YBiCLJQQ3"},"Id":"0172A0000YBiCLJQQ3","ParentId":"a9C2A000000XZclUAG","CreatedDate":"2016-12-21T21:56:39.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000ZmpbvtQQA"},"Id":"0172A0000ZmpbvtQQA","ParentId":"a9C2A000000blcLUAQ","CreatedDate":"2017-05-05T17:48:13.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000uvNIO2QAO"},"Id":"0171Y0000uvNIO2QAO","ParentId":"a9C1Y0000004E8uUAE","CreatedDate":"2020-01-22T23:35:32.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000w9amIaQAI"},"Id":"0171Y0000w9amIaQAI","ParentId":"a9C1Y0000004EBtUAM","CreatedDate":"2020-02-27T19:19:32.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000skFQNgQAO"},"Id":"0171Y0000skFQNgQAO","ParentId":"a9C1Y0000004E5RUAU","CreatedDate":"2019-11-23T23:07:46.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000tTJipbQAD"},"Id":"0171Y0000tTJipbQAD","ParentId":"a9C1Y0000004E73UAE","CreatedDate":"2019-12-16T15:32:34.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000byaXfYQAU"},"Id":"0170Z0000byaXfYQAU","ParentId":"a9C0Z0000004E5ZUAU","CreatedDate":"2018-06-07T12:38:59.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000bdOClFQAW"},"Id":"0170Z0000bdOClFQAW","ParentId":"a9C0Z000000TNhVUAW","CreatedDate":"2018-05-22T17:54:46.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00013PHXeUAAX"},"Id":"0174z00013PHXeUAAX","ParentId":"a9C4z000000oLm1EAE","CreatedDate":"2021-05-13T18:32:54.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00016Q2a4kAAB"},"Id":"0174z00016Q2a4kAAB","ParentId":"a9C4z000000wk2nEAA","CreatedDate":"2021-08-06T17:11:22.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000b3ZRCIQA4"},"Id":"0170Z0000b3ZRCIQA4","ParentId":"a9C0Z000000TNeqUAG","CreatedDate":"2018-04-24T15:08:06.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000nPPGxxQAH"},"Id":"0171Y0000nPPGxxQAH","ParentId":"a9C1Y000000fxVFUAY","CreatedDate":"2019-04-29T18:30:29.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000oE0KENQA3"},"Id":"0171Y0000oE0KENQA3","ParentId":"a9C1Y000000fxWSUAY","CreatedDate":"2019-05-31T19:52:42.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00013p96vnAAA"},"Id":"0174z00013p96vnAAA","ParentId":"a9C4z000000oLmQEAU","CreatedDate":"2021-05-25T23:29:21.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00013rWM4XAAW"},"Id":"0174z00013rWM4XAAW","ParentId":"a9C4z000000oLmVEAU","CreatedDate":"2021-05-27T00:38:09.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00011OhyQdAAJ"},"Id":"0174z00011OhyQdAAJ","ParentId":"a9C4z000000oLkPEAU","CreatedDate":"2021-03-07T17:46:32.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00014P6WM4QAN"},"Id":"0171Y00014P6WM4QAN","ParentId":"a9C1Y0000008PepUAE","CreatedDate":"2020-11-25T17:56:48.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000XGw8wXQQR"},"Id":"0172A0000XGw8wXQQR","ParentId":"a9C2A000000PBaBUAW","CreatedDate":"2016-10-10T16:49:59.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000PhcK6DIQU"},"Id":"017F00000PhcK6DIQU","ParentId":"a9CF00000008OM2MAM","CreatedDate":"2016-01-04T18:09:55.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000VRWdLBIQ1"},"Id":"017F00000VRWdLBIQ1","ParentId":"a9CF00000008P6PMAU","CreatedDate":"2016-07-28T23:59:57.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JVVNyzIQH"},"Id":"017F00000JVVNyzIQH","ParentId":"a9CF0000000PBAcMAO","CreatedDate":"2015-04-17T17:55:31.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JVVNz0IQH"},"Id":"017F00000JVVNz0IQH","ParentId":"a9CF0000000PBAdMAO","CreatedDate":"2015-04-17T17:55:31.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000OKYxvuIQD"},"Id":"017F00000OKYxvuIQD","ParentId":"a9CF0000000blT4MAI","CreatedDate":"2015-11-05T23:13:45.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000cDinMDQAZ"},"Id":"0172A0000cDinMDQAZ","ParentId":"a9C2A000000Kz15UAC","CreatedDate":"2017-11-10T23:54:33.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000UcibJsIQI"},"Id":"017F00000UcibJsIQI","ParentId":"a9CF00000008P3VMAU","CreatedDate":"2016-07-01T09:50:21.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000exGsX1QAK"},"Id":"0170Z0000exGsX1QAK","ParentId":"a9C0Z0000008PVOUA2","CreatedDate":"2018-10-15T23:09:28.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YyEfV3QQK"},"Id":"0172A0000YyEfV3QQK","ParentId":"a9C2A000000XZkGUAW","CreatedDate":"2017-03-02T22:16:34.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000dPqFUHQA3"},"Id":"0172A0000dPqFUHQA3","ParentId":"a9C2A000000KzXkUAK","CreatedDate":"2018-02-01T17:01:35.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000X3mlCnQQI"},"Id":"0172A0000X3mlCnQQI","ParentId":"a9C2A000000PBYeUAO","CreatedDate":"2016-09-22T13:12:00.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000ayVnJsQAK"},"Id":"0172A0000ayVnJsQAK","ParentId":"a9C2A000000bmESUAY","CreatedDate":"2017-08-04T12:52:51.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00012kTWhvQAG"},"Id":"0171Y00012kTWhvQAG","ParentId":"a9C1Y0000008Pa9UAE","CreatedDate":"2020-09-28T22:31:07.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000bzbOt9QAE"},"Id":"0172A0000bzbOt9QAE","ParentId":"a9C2A000000Kz0WUAS","CreatedDate":"2017-10-25T13:24:12.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000Zv44fQQQQ"},"Id":"0172A0000Zv44fQQQQ","ParentId":"a9C2A000000blrzUAA","CreatedDate":"2017-05-16T23:38:27.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000nqCB9VQAW"},"Id":"0171Y0000nqCB9VQAW","ParentId":"a9C1Y000000fxVtUAI","CreatedDate":"2019-05-15T23:31:02.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000N7c11cIQA"},"Id":"017F00000N7c11cIQA","ParentId":"a9CF0000000XZIGMA4","CreatedDate":"2015-09-16T21:19:06.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000O1YFpfIQG"},"Id":"017F00000O1YFpfIQG","ParentId":"a9CF0000000blP2MAI","CreatedDate":"2015-10-23T19:05:05.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000KmSvSkIQK"},"Id":"017F00000KmSvSkIQK","ParentId":"a9CF0000000PBJAMA4","CreatedDate":"2015-06-11T01:25:08.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000MKCfqsIQD"},"Id":"017F00000MKCfqsIQD","ParentId":"a9CF0000000PBU3MAO","CreatedDate":"2015-08-14T20:34:26.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000pBClqEQAT"},"Id":"0171Y0000pBClqEQAT","ParentId":"a9C1Y000000fxgDUAQ","CreatedDate":"2019-07-11T18:18:41.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y000173oG0DQAU"},"Id":"0171Y000173oG0DQAU","ParentId":"a9C1Y0000008PhPUAU","CreatedDate":"2021-02-20T22:18:37.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000xbrSa4QAE"},"Id":"0171Y0000xbrSa4QAE","ParentId":"a9C1Y0000004EI1UAM","CreatedDate":"2020-04-15T22:31:26.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00014p7NOiAAM"},"Id":"0174z00014p7NOiAAM","ParentId":"a9C4z000000oLnJEAU","CreatedDate":"2021-06-24T04:10:58.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000X3Nwe4QQC"},"Id":"0172A0000X3Nwe4QQC","ParentId":"a9C2A000000PBYZUA4","CreatedDate":"2016-09-21T21:25:54.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000w55UHdQAM"},"Id":"0171Y0000w55UHdQAM","ParentId":"a9C1Y0000004EBeUAM","CreatedDate":"2020-02-25T21:29:54.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YOaZuLQQV"},"Id":"0172A0000YOaZuLQQV","ParentId":"a9C2A000000XZe3UAG","CreatedDate":"2017-01-11T17:10:13.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000q9LLKbQAO"},"Id":"0171Y0000q9LLKbQAO","ParentId":"a9C1Y000000fxh6UAA","CreatedDate":"2019-08-19T19:23:57.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00013ciFnJQAU"},"Id":"0171Y00013ciFnJQAU","ParentId":"a9C1Y0000008Pd8UAE","CreatedDate":"2020-10-27T22:14:05.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000ahD4bJQAS"},"Id":"0172A0000ahD4bJQAS","ParentId":"a9C2A000000bmBsUAI","CreatedDate":"2017-07-11T13:24:32.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00016YKJWxQAP"},"Id":"0171Y00016YKJWxQAP","ParentId":"a9C1Y0000008PgbUAE","CreatedDate":"2021-02-03T16:19:08.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y00016YHY6AQAX"},"Id":"0171Y00016YHY6AQAX","ParentId":"a9C1Y0000008PgWUAU","CreatedDate":"2021-02-03T00:33:39.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000UpbR6fIQE"},"Id":"017F00000UpbR6fIQE","ParentId":"a9CF00000008P4dMAE","CreatedDate":"2016-07-08T18:58:54.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000aMdrFpQAJ"},"Id":"0172A0000aMdrFpQAJ","ParentId":"a9C2A000000bmApUAI","CreatedDate":"2017-06-26T22:24:46.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000MKCXnCIQX"},"Id":"017F00000MKCXnCIQX","ParentId":"a9CF0000000PBToMAO","CreatedDate":"2015-08-14T20:32:12.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000MKCZuvIQH"},"Id":"017F00000MKCZuvIQH","ParentId":"a9CF0000000PBTtMAO","CreatedDate":"2015-08-14T20:32:45.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000MKCaN6IQL"},"Id":"017F00000MKCaN6IQL","ParentId":"a9CF0000000PBTyMAO","CreatedDate":"2015-08-14T20:34:02.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000MKBPzvIQH"},"Id":"017F00000MKBPzvIQH","ParentId":"a9CF0000000PBTjMAO","CreatedDate":"2015-08-14T19:54:29.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000X2AyT4QQK"},"Id":"0172A0000X2AyT4QQK","ParentId":"a9C2A000000PBYUUA4","CreatedDate":"2016-09-20T18:52:14.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000XanMc9QQE"},"Id":"0172A0000XanMc9QQE","ParentId":"a9C2A000000XZWTUA4","CreatedDate":"2016-10-27T23:19:41.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000XjOQXhQQO"},"Id":"0172A0000XjOQXhQQO","ParentId":"a9C2A000000XZX2UAO","CreatedDate":"2016-11-10T19:09:55.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000YD9IfHQQV"},"Id":"0172A0000YD9IfHQQV","ParentId":"a9C2A000000XZdZUAW","CreatedDate":"2016-12-23T17:34:39.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000Y5q6tBQQQ"},"Id":"0172A0000Y5q6tBQQQ","ParentId":"a9C2A000000XZb9UAG","CreatedDate":"2016-12-16T14:50:00.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0171Y0000t3TnLnQAK"},"Id":"0171Y0000t3TnLnQAK","ParentId":"a9C1Y0000004E65UAE","CreatedDate":"2019-12-04T21:07:44.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00011ayXnyAAE"},"Id":"0174z00011ayXnyAAE","ParentId":"a9C4z000000oLkeEAE","CreatedDate":"2021-03-15T20:52:55.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000duuLFgQAM"},"Id":"0172A0000duuLFgQAM","ParentId":"a9C2A000000PBipUAG","CreatedDate":"2018-03-08T01:24:58.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00019Ku9UwAAJ"},"Id":"0174z00019Ku9UwAAJ","ParentId":"a9C4z000000wk5IEAQ","CreatedDate":"2021-10-27T12:46:30.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00019joFtjAAE"},"Id":"0174z00019joFtjAAE","ParentId":"a9C4z000000wk5cEAA","CreatedDate":"2021-11-08T20:48:44.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z000177ryxaAAA"},"Id":"0174z000177ryxaAAA","ParentId":"a9C4z000000wk32EAA","CreatedDate":"2021-08-26T03:09:49.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00017LER0wAAH"},"Id":"0174z00017LER0wAAH","ParentId":"a9C4z000000wk3CEAQ","CreatedDate":"2021-08-31T18:47:06.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00018tHppcAAC"},"Id":"0174z00018tHppcAAC","ParentId":"a9C4z000000wk4eEAA","CreatedDate":"2021-10-15T02:49:23.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00018aDh3ZAAS"},"Id":"0174z00018aDh3ZAAS","ParentId":"a9C4z000000wk4KEAQ","CreatedDate":"2021-10-05T11:13:00.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000dd2OQ9QAM"},"Id":"0172A0000dd2OQ9QAM","ParentId":"a9C2A000000PBhNUAW","CreatedDate":"2018-02-16T00:14:24.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00017F4MdCAAV"},"Id":"0174z00017F4MdCAAV","ParentId":"a9C4z000000wk37EAA","CreatedDate":"2021-08-29T03:45:00.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000e6vxu3QAA"},"Id":"0172A0000e6vxu3QAA","ParentId":"a9C2A0000004CSnUAM","CreatedDate":"2018-03-20T22:11:19.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000RQZEn4IQH"},"Id":"017F00000RQZEn4IQH","ParentId":"a9CF00000008OZQMA2","CreatedDate":"2016-03-09T23:30:15.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0174z00016VnEr8AAF"},"Id":"0174z00016VnEr8AAF","ParentId":"a9C4z000000wk2sEAA","CreatedDate":"2021-08-09T21:41:52.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000MfoqasIQA"},"Id":"017F00000MfoqasIQA","ParentId":"a9CF0000000XZARMA4","CreatedDate":"2015-08-29T02:50:22.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000JbZMPBIQ4"},"Id":"017F00000JbZMPBIQ4","ParentId":"a9CF0000000PBBaMAO","CreatedDate":"2015-04-22T15:30:23.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000SifR7IIQU"},"Id":"017F00000SifR7IIQU","ParentId":"a9CF00000008OihMAE","CreatedDate":"2016-04-23T00:36:14.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000Trj366IQA"},"Id":"017F00000Trj366IQA","ParentId":"a9CF00000008OvlMAE","CreatedDate":"2016-06-06T13:06:18.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000Trktn8IQA"},"Id":"017F00000Trktn8IQA","ParentId":"a9CF00000008OvqMAE","CreatedDate":"2016-06-06T18:30:27.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000V8IuOqIQK"},"Id":"017F00000V8IuOqIQK","ParentId":"a9CF00000008P5MMAU","CreatedDate":"2016-07-18T15:38:50.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000PmIKP9IQO"},"Id":"017F00000PmIKP9IQO","ParentId":"a9CF00000008OMlMAM","CreatedDate":"2016-01-07T03:43:54.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000PCKAgiIQH"},"Id":"017F00000PCKAgiIQH","ParentId":"a9CF0000000KyuYMAS","CreatedDate":"2015-12-13T05:50:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000P7rOOpIQM"},"Id":"017F00000P7rOOpIQM","ParentId":"a9CF0000000KyuEMAS","CreatedDate":"2015-12-10T01:08:25.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000eAYE98QAH"},"Id":"0170Z0000eAYE98QAH","ParentId":"a9C0Z0000008PUVUA2","CreatedDate":"2018-09-18T00:58:30.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000dq7TZKQA2"},"Id":"0170Z0000dq7TZKQA2","ParentId":"a9C0Z0000008PTwUAM","CreatedDate":"2018-09-04T21:57:16.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0172A0000XE7EB3QQN"},"Id":"0172A0000XE7EB3QQN","ParentId":"a9C2A000000PBZwUAO","CreatedDate":"2016-10-04T18:05:36.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/017F00000R9qSHsIQM"},"Id":"017F00000R9qSHsIQM","ParentId":"a9CF00000008OWqMAM","CreatedDate":"2016-02-29T21:27:48.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000fAxKbjQAF"},"Id":"0170Z0000fAxKbjQAF","ParentId":"a9C0Z0000008PViUAM","CreatedDate":"2018-10-24T17:18:45.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000drgXEkQAM"},"Id":"0170Z0000drgXEkQAM","ParentId":"a9C0Z0000008PU1UAM","CreatedDate":"2018-09-05T18:01:55.000+0000","Field":"created","OldValue":null,"NewValue":null},{"attributes":{"type":"Incident__History","url":"/services/data/v53.0/sobjects/Incident__History/0170Z0000fcb9sbQAA"},"Id":"0170Z0000fcb9sbQAA","ParentId":"a9C0Z0000008PW7UAM","CreatedDate":"2018-11-11T23:41:45.000+0000","Field":"created","OldValue":null,"NewValue":null}][{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKmdEAH"},"Id":"a1P4z00000BAKmdEAH","CreatedDate":"2021-11-08T21:12:38.000+0000","IncidentId__c":"a9C4z000000wk5cEAA","UpdateLog__c":"Okta is experiencing an issue where admins may be experiencing errors while utilizing Workflow designer and invoking flows. Scheduled flows are not affected and continue to run without issue. Engineering is currently investigating the issue. We will provide an update within the next 30 minutes","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKmiEAH"},"Id":"a1P4z00000BAKmiEAH","CreatedDate":"2021-11-08T21:51:19.000+0000","IncidentId__c":"a9C4z000000wk5cEAA","UpdateLog__c":"Okta is experiencing an issue where admins may be experiencing errors while utilizing Workflow designer and invoking flows. In addition, event hook delivery will also be delayed. Scheduled flows are not affected and continue to run without issue. Engineering is currently investigating the issue. We will provide an update within the next 30 minutes","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKn7EAH"},"Id":"a1P4z00000BAKn7EAH","CreatedDate":"2021-11-09T00:36:19.000+0000","IncidentId__c":"a9C4z000000wk5cEAA","UpdateLog__c":"Okta has confirmed recovery of Workflow Console, flow invocation, and all services are running without any issues. The hook backlog has been eliminated and the workflows system has returned to normal status. Additional root cause information will be provided.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKmEEAX"},"Id":"a1P4z00000BAKmEEAX","CreatedDate":"2021-10-27T22:14:57.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Resolved: The SMS MFA delivery issue caused by AT&T network issues has been resolved. During this event a subset of users who were using SMS via AT&T may not have received the response to their MFA requests. Additional root cause information will be provided.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKm9EAH"},"Id":"a1P4z00000BAKm9EAH","CreatedDate":"2021-10-27T21:41:18.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Our monitoring shows a return to normal conditions with MFA for customers that utilize SMS with AT&T. However, we have yet to receive confirmation from our providers that the issue is fully resolved. Our next update will be in one hour or sooner if additional information becomes available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1xWbEAJ"},"Id":"a1P4z00000B1xWbEAJ","CreatedDate":"2021-06-25T16:24:52.000+0000","IncidentId__c":"a9C4z000000oLnJEAU","UpdateLog__c":"Root Cause Analysis: \r\n\r\nWe sincerely apologize for any impact this incident may have caused to you and your business. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding a recent incident that may have disrupted your ability to use the Okta service. We are committed to implementing additional improvements to the service to prevent future occurrences of this incident.\r\n\r\n \r\n\r\nTiming:\r\n\r\nAt 8:50pm Okta detected the issue. We deployed a correction at 9:09pm. All services were confirmed fully restored by 9:30pm.\r\n\r\n \r\n\r\nDetection and Impact\r\n\r\nOn June 23rd at approximately 8:50PM (PT) during a planned maintenance operation Okta detected intermittent request failures for some customers in US Cell 6. Investigation revealed that some requests to the Okta service resulted in 50X errors or slowness. Further analysis determined that the success rate of requests was highly intermittent during the period with >74% success at some instances in time, and <20% during others.\r\n\r\n \r\n\r\nRemediation Steps\r\n\r\nImmediately upon determining a resolution path, Okta began to address the impact and restore service availability. At 9:09PM Okta began to put http proxies into rotation to support customer traffic which restored most functionality. By 9:30PM all services were confirmed fully restored.\r\n\r\n \r\n\r\nRoot Cause Summary\r\n\r\nOkta Determined that this incident occurred during a planned maintenance operation that updates http proxies in US Cell 6. During the execution of this operation two things contributed to the root cause:\r\n\r\nThe procedure is intended to incrementally update http proxy groups in sequential fashion. During this incident, the grouping of http proxies was skipped which caused more http proxies to be taken out of rotation in US Cell 6 than intended. \r\n\r\nThe automated validation which assures that a sufficient number of http proxies are available and in rotation, was mistakenly disabled during this maintenance operation.\r\n\r\nHad only one of these mistakes occurred, the automated operation would have prevented the http proxy maintenance from even starting in US Cell 6.\r\n\r\nPreventive Actions\r\n\r\nOkta is taking multiple steps to prevent recurrence of this issue:\r\n\r\nImmediately stop the use of the operational procedure that led to the incident\r\n\r\nUpdate the standard operating procedure to ensure additional reviews are conducted\r\n\r\nImprove automation to make the http proxy capacity check mandatory in order to eliminate cell wide impact when the proxy infrastructure requires updates\r\n\r\nIn addition to the steps above, Okta has implemented a general moratorium for US Cell 6, as well as a moratorium on http proxy operations across all cells until the identified preventative measures and automations are fully implemented and verified.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKiMEAX"},"Id":"a1P4z00000BAKiMEAX","CreatedDate":"2021-08-31T19:51:31.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We are mitigating the increased error rates observed in US Cells 5, 7, 12, and 14, and Preview 3 and will provide further status updates as soon as we have more information.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1xVYEAZ"},"Id":"a1P4z00000B1xVYEAZ","CreatedDate":"2021-05-28T22:12:47.000+0000","IncidentId__c":"a9C4z000000oLmVEAU","UpdateLog__c":"Root Cause Analysis:\r\n\r\nAt 11:21am PST on May 25th, Okta received a customer support request regarding user provisioning within the RingCentral Application. Evaluation of the support request determined that provisioning operations and password synchronization between the RingCentral Application and the Okta Service was not functioning as expected.\r\n\r\nInvestigation of this issue revealed an unexpected behavior while building the request URL path for most of the API calls in the RingCentral provisioning flow, which caused provisioning operations to fail. Further investigation determined the unexpected behavior was a result of a client library update, which had introduced a breaking change for the provisioning operations flow of the RingCentral application. Users attempting to authenticate into RingCentral via the Okta service were not impacted, however, user administrators attempting to provision or deprovision users within the RingCentral application would have been unable to complete profile updates during this time.\r\n\r\nAt 5:30 PM PST on May 26th Okta reverted to the prior release of the affected service and RingCentral provisioning operations were observed to complete as expected. At 6:45 PM PST the revert had been successfully deployed to all cells and the impacted application was able to synchronize with Okta.\r\n\r\nAs part of Okta’s commitment to integration integrity, a manual audit to proactively identify other applications that could be impacted by similar client library updates was completed alongside a review of automated test cases to ensure similar issues are detected in the future.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKkXEAX"},"Id":"a1P4z00000BAKkXEAX","CreatedDate":"2021-10-07T17:49:15.000+0000","IncidentId__c":"a9C4z000000wk4KEAQ","UpdateLog__c":"We sincerely apologize for any impact this incident has caused to you and your business and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident.\r\n\r\nDetection and Impact:\r\nOn October 4th at approximately 9:23AM (PDT) Okta received customer reports that they were receiving Invalid Assertion SAML errors in Amazon Web Services (AWS) when trying to Single Sign On (SSO) using our AWS Account Federation app. The issue was observed to impact orgs with provisioning API enabled but provisioning options - Create Users and Update User Attributes - disabled on all Cells. \r\n \r\nRemediation Steps:\r\nThe reports stemmed from an upgrade to the integration which triggered app meta-data migrations for customers using the app which started on October 4th at approximately 7:50AM (PDT). At 4:53AM (PDT) on October 5th, the root cause was confirmed and Okta conducted a rollback of the app version on all Cells. Rollbacks were completed by 6:05AM (PDT) when the service was fully restored and operated as expected. \r\n \r\nRoot Cause Summary:\r\nPrior to the AWS Account Federation app upgrade, a code change had been made to how provisioning options were handled when migrating customers to a new app version. This change resulted in AWS roles not being migrated for customers who had enabled provisioning, but had disabled all provisioning options. This omission of the role meta-data caused SSO into AWS to fail.\r\n \r\nPreventative Actions: \r\nOkta is working on a fix to the application migration process so that the role information will be properly added to the SAML token. Additional testing and monitoring are being added to guard against a recurrence of this issue in the future.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKhsEAH"},"Id":"a1P4z00000BAKhsEAH","CreatedDate":"2021-08-23T18:00:33.000+0000","IncidentId__c":"a9C4z000000wk2nEAA","UpdateLog__c":"Root Cause Analysis:\r\nWe sincerely apologize for any impact this incident may have caused to you and your business. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding a recent incident that may have disrupted your ability to use the Okta service. We are committed to implementing additional improvements to the service to prevent future occurrences of this incident.\r\n\r\nTiming:\r\nOn August 6th at 5:27AM PDT, customers began reporting intermittent delays for SMS MFA to US based numbers. At 6:56AM Okta began mitigating the issue to move the Cells to the failover provider. The mitigation procedures completed at 7:22AM.\r\n\r\nDetection and Impact:\r\nIn all Cells, some customers who leverage SMS as an authentication factor experienced intermittent delivery delays on their first request to US based phone numbers. The end users’ retry requests utilized Okta’s failover provider and SMS authentications were successfully received.  \r\n\r\nOkta confirmed the issue was with its downstream primary provider by reviewing information posted on the provider’s status page.\r\n\r\nRoot Cause Summary:\r\nThe primary SMS provider experienced connection instability issues with one of its downstream providers.  \r\n\r\nRemediation Steps:\r\nOkta mitigated the issue by swapping the primary SMS provider control to the failover provider. \r\n\r\nPreventive Actions:\r\nOkta is taking steps to improve the speed of detection and implementation of corrective action.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKilEAH"},"Id":"a1P4z00000BAKilEAH","CreatedDate":"2021-08-31T22:39:30.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We have mitigated all observed errors and issues in the US-West region affecting US cells 5, 7, 12, 14, and Preview 3. Error rates have completely subsided and the service has returned to normal status, queued events are progressing and should complete in the next few hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKnMEAX"},"Id":"a1P4z00000BAKnMEAX","CreatedDate":"2021-11-11T01:47:05.000+0000","IncidentId__c":"a9C4z000000wk5cEAA","UpdateLog__c":"Root Cause Analysis\r\n\r\nWe sincerely apologize for any impact this incident has caused you, your business or your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident.\r\n\r\nDetection and Impact\r\n\r\nOn November 9, 2021 at 12:09 pm (PST), Okta monitoring began receiving alerts of failed Workflows execution requests on US Cells 1, 2, 3, 4, 6, 7 and 11 impacting Workflows services. At the same time, Okta began receiving reports from customers who were impacted in the following scenarios:\r\nWorkflows Console - Unavailable from 12:45 pm - 3:50 pm\r\nWorkflow API calls experienced timeouts from 12:55 pm - 2:25 pm. Some API calls began to succeed at 2:25 pm until full recovery at 3:50 pm\r\nEvent Hooks processing was delayed by cell per the following time windows\r\n\r\nUS Cell 1: 1:50 PM PST - 3:20 PM PST\r\n\r\nUS Cell 2: 1:05 PM PST - 4:47 PM PST\r\n\r\nUS Cell 3: 1:28 PM PST - 4:15 PM PST\r\n\r\nUS Cell 4: 1:15 PM PST - 3:25 PM PST\r\n\r\nUS Cell 6: 1:10 PM PST - 6:00 PM PST\r\n\r\nUS Cell 7: 1:00 PM PST - 4:20 PM PST\r\n\r\nUS Cell 11: limited impact\r\n\r\nWorkflows scheduled flows were not impacted. \r\n\r\nRoot Cause Summary\r\n\r\nOkta determined that the root cause was a maintenance update which required a restart of some Workflows platform services. The restart, combined with system load on these services, impacted the orchestration between the services which in turn incorrectly caused healthy worker servers to be marked as unhealthy. This caused these worker servers to be taken out of service, which propagated the impact and caused unavailability of Workflows Console and Event Hook processors.\r\n\r\nRemediation Steps\r\n\r\nOkta took action to identify the affected platform services, and restarted these services to restore them to health adding additional capacity to aid in faster recovery. At 3:02 pm PST, the services started to recover and normal operation was restored at 3:50 pm PST. Event Hooks that had been triggered during the impact window were queued and continued redelivery to get current through 6:00 pm PST (see catchup time windows per cell above).\r\n\r\nPreventive Actions\r\n\r\nTo mitigate the impact to Workflows for similar future events, Okta will implement configuration and process improvements for these types of maintenance updates. Okta is also implementing additional monitoring and alerting to further reduce the time needed for issue identification and resolution.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1wSOEAZ"},"Id":"a1P4z00000B1wSOEAZ","CreatedDate":"2021-03-07T18:30:27.000+0000","IncidentId__c":"a9C4z000000oLkPEAU","UpdateLog__c":"We continue to investigate the intermittent issues with external network connectivity in the OK8 (APAC) region. Depending on location, users may not be able to access our service. We are actively investigating the issue and will provide a status update as soon as we have further information.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1xVTEAZ"},"Id":"a1P4z00000B1xVTEAZ","CreatedDate":"2021-05-27T23:56:02.000+0000","IncidentId__c":"a9C4z000000oLmQEAU","UpdateLog__c":"Root Cause Analysis: On May 25th 2021 at 11:03 AM PDT, Okta was alerted about health check failures in US Cell 6 in the load balancer tier. Upon investigation, it was determined that a recent increase in capacity generated a corresponding increase in health check requests that were consuming connections in router queues and reducing the ability for additional connections to be established.\r\n\r\nIn order to minimize customer disruption, at 11:25 AM Okta Engineering added additional router capacity to US Cell 6 and began migrating customer orgs to a redundant router stack.\r\n\r\nAt 11:47 AM, Okta Engineering took additional action to reduce the load balancing tier health check frequency from every 10 seconds to 30 seconds, which reduced the number of connection requests and allowed the system to recover.\r\n\r\nThroughout the service disruption window, 90% of authentication requests continued to successfully authenticate, but customers who were impacted would have observed intermittent failure of API requests, increased latency, required a retry, or may have experienced a 504 gateway timeout error.\r\n\r\nAs a result of this event, Okta has taken the action to audit monitoring alerts to ensure that secondary automated alerts reflect the proper thresholds for notification that will enable swift action should this issue arise in the future.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKlVEAX"},"Id":"a1P4z00000BAKlVEAX","CreatedDate":"2021-10-27T13:58:32.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"We are seeing some recovery of the AT&T service and increase in SMS success rates. We continue to monitor and are seeing improvement","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1xV4EAJ"},"Id":"a1P4z00000B1xV4EAJ","CreatedDate":"2021-05-15T14:22:57.000+0000","IncidentId__c":"a9C4z000000oLm1EAE","UpdateLog__c":"Root Cause Analysis:\r\nOn May 13, 2021 at 10:43AM PDT Okta began receiving alerts for an abnormally high volume of network traffic on US Cell 6. Shortly thereafter, Okta began receiving reports regarding service availability impact from customers in US Cell 6. At 10:50AM Okta’s infrastructure provider began to automatically limit the volume of traffic to US Cell 6 to mitigate impact to downstream services. This limitation dropped a percentage of all network traffic, including some requests from customers. Okta customers using Custom Domains or Mobile scenarios were not affected.\r\n\r\nAt 11:03AM, Okta contacted our infrastructure provider to increase throughput capacity and allow more traffic. Concurrently, Okta added additional HTTP proxies in US Cell 6 and the service returned to normal operation by 11:44AM.\r\n\r\nWe sincerely apologize for the impact to affected customers and are continuously taking steps to improve the Okta service and mitigate the impact of similar future events, including:\r\n\r\nImproving our load balancer configurations to not send traffic to degraded zones\r\n\r\nWorking with our infrastructure provider to set reasonable rate limits and network traffic capacity","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKi7EAH"},"Id":"a1P4z00000BAKi7EAH","CreatedDate":"2021-08-28T07:05:35.000+0000","IncidentId__c":"a9C4z000000wk32EAA","UpdateLog__c":"We sincerely apologize for any impact this incident has caused to you and your business and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident.\r\n\r\n\r\nDetection and Impact\r\n\r\nOn August 25th at approximately 6:03PM (PT) Okta received customer reports that they were receiving SSL certificate warnings on their clients for requests to the Okta service. Investigation revealed that this issue impacted custom URL domain users on US Cell 12. In addition, this issue was observed to occur intermittently during this period with the overall success rate of greater than 75% for users. Overall response time and Admin requests were not impacted during this period.\r\n\r\n\r\nRoot Cause Summary\r\n\r\nOkta determined that the root cause was a maintenance operation to update routers in US Cell 12. During the initial execution of this operation, the custom domain SSL certificates failed to properly load on a router which prevented establishing TLS connections for some custom domains intermittently by displaying a warning to users. The impact was intermittent because requests are handled by a pool of routers, and only requests received by the affected member of the US Cell12 router pool were impacted.\r\n\r\n\r\nRemediation Steps\r\n\r\nAt 8:10PM, Okta began to address the impact by removing the affected router that did not have the SSL certificates loaded from the router group. Following resumption of normal operation, Okta successfully added a router with correct certificates back into the router group for US Cell12.\r\n\r\n\r\nPreventive Actions\r\n\r\nOkta is improving the process for deploying and verifying SSL certificates as well as adding detection measures to prevent occurence of this issue in the future.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKjKEAX"},"Id":"a1P4z00000BAKjKEAX","CreatedDate":"2021-09-03T03:23:48.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We sincerely apologize for any impact this incident has caused to you and your business and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident.\r\n\r\nDetection and Impact:\r\n\r\nOn August 31st at 10:59AM (PT) Okta detected an increased error rate across US Cells 5, 7, 12, 14 and Workflows FL5, 12, 14. At 2:33PM Okta also detected an increased error rate in EU Cell 2.\r\n\r\nDuring the incident, customers may have experienced intermittent errors when attempting to access the service. Success rates for customer requests averaged ~85% and in most cases, a retry operation would succeed. Error rates decreased as Okta implemented mitigation measures, but the overall durations of the incident are as follows:\r\n\r\nUS Cells 5, 7, 12, 14 from 10:59AM to 3:35PM (PDT) \r\n\r\nEU Cell 2 from 2:15PM to 3:30PM (PDT).\r\n\r\nWorkflow cells FL5, FL12 and FL14 Workflows 11:02AM - 12:28PM (PDT) \r\n\r\nPlease note that there were use cases and periods where some customers experienced significantly higher error rates and request latency, for example requests that initiated outbound calls, that are not captured in the all up analysis across all impacted tenants.\r\n\r\nRoot Cause Summary:\r\n\r\nThe root cause of the incident is one of Okta’s cloud service providers experienced network issues in one availability zone in one of their US locations and one availability zone in one of their European locations which caused inbound and outbound connection errors that impacted the Okta service. \r\n\r\nBecause of the intermittent nature of the errors, our service did not automatically take impacted tiers out of service. This meant they continued to accept requests even though they might not be able to successfully execute them instead of leaving them to the unimpacted tiers in the other two availability zones that cells are hosted across. Okta Engineering had to engage to manually mitigate and resolve the issues. \r\n\r\nRemediation Steps:\r\n\r\nOnce the issue with the Okta Cloud Service Provider was confirmed, Okta began to divert traffic to unaffected infrastructure beginning at 11:47AM. At approximately 3:35PM, the service was restored and operating as expected.\r\n\r\nPreventive Actions:\r\n\r\nOkta is working closely with the provider to investigate mitigations and improve response to these types of incidences. Additionally, Okta is reviewing procedures to improve service resiliency actions in degraded data center scenarios. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1wSdEAJ"},"Id":"a1P4z00000B1wSdEAJ","CreatedDate":"2021-03-07T19:50:47.000+0000","IncidentId__c":"a9C4z000000oLkPEAU","UpdateLog__c":"Okta experienced a service disruption in the OK8 cell (APAC) where customers may have experienced intermittent network connectivity due to a service provider issue. Service has returned to normal as of 11:30 AM PST. We are working with the service provider to investigate the root cause of this service disruption and will post the RCA to https://status.okta.com within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKlaEAH"},"Id":"a1P4z00000BAKlaEAH","CreatedDate":"2021-10-27T14:34:34.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"We are seeing some recovery of the AT&T service and increase in SMS success rates. We continue to monitor and are seeing improvement","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKcRUAU"},"Id":"a1P1Y000007zKcRUAU","CreatedDate":"2020-02-28T00:53:07.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta is continuing to work towards a fix that will resolve this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKcWUAU"},"Id":"a1P1Y000007zKcWUAU","CreatedDate":"2020-02-28T01:51:35.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta is continuing to work towards a fix that will resolve this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKcbUAE"},"Id":"a1P1Y000007zKcbUAE","CreatedDate":"2020-02-28T02:54:36.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta is continuing to work towards a fix that will resolve this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKcgUAE"},"Id":"a1P1Y000007zKcgUAE","CreatedDate":"2020-02-28T04:42:31.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta is continuing to work towards a fix that will resolve this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKclUAE"},"Id":"a1P1Y000007zKclUAE","CreatedDate":"2020-02-28T05:28:50.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Resolved: Okta has successfully deployed and verified a fix for customers whose previous configurations were affected as a result of this issue. Additional root cause information will be provided within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKd0UAE"},"Id":"a1P1Y000007zKd0UAE","CreatedDate":"2020-03-03T06:08:22.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Root Cause Analysis: On February 27th at approximately 8:15am PST, Okta received a customer report of an error in US Cell 1 when attempting to add an additional AWS account. Upon investigation, Okta found that customers in all cells would experience errors if attempting to add child AWS accounts or modify existing AWS integration configurations.\r\n\r\nIn order to prevent further issues to customer configurations, Okta immediately made all AWS app instance configurations read-only. This change was rolled out to all cells by 1:41pm PST until the root cause of the authentication issues could be identified and resolved.\r\n\r\nFurther investigation determined that the errors were the result of a recent upgrade to the AWS SDK which conflicted with Okta's existing logic for resolving STS endpoints. In the majority of configurations, Okta's logic would previously default to the Global STS endpoint, however after the upgrade the same resolution process began always returning the lexicographically first endpoint, which was the STS endpoint for the Asia Pacific region (Hong Kong). Because the Version 1 STS tokens that Okta generates are only valid in regions which are enabled by default, and Asia Pacific is not enabled by default, Okta's subsequent AssumeRole calls to that region were failing.\r\n\r\nThough the SDK upgrade was tested in some environments, the affected configuration workflow had not been incorporated into any automated or manual end-to-end testing flows; this service interruption highlighted a gap in our regression test plan within the AWS application integration.\r\nOnce Okta determined the cause of the errors, a fix was implemented to route all STS calls to the global endpoint. This fix was deployed to US Cell 1 at 8:49pm PST and then to all cells. By February 28 at 9:01pm PST, the update was completed and service was fully restored.\r\n\r\nOkta is currently working to update how endpoints and regions are passed to the AWS SDK to increase efficiency and reduce latency. As part of our on-going focus on reliability, Okta is also actively working to identify and audit test cases where application integrations and core infrastructure rely on common elements to prevent issues like this from occurring in the future.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKgEUAU"},"Id":"a1P1Y000007zKgEUAU","CreatedDate":"2020-03-16T19:41:42.000+0000","IncidentId__c":"a9C1Y0000004EBeUAM","UpdateLog__c":"Root Cause Analysis: On February 25th at approximately ‪1:05pm PST, Okta’s monitoring provided alerts that some import user jobs on Okta Preview 1, US Cell 6, and US Cell 7 were experiencing longer than normal wait times.\r\nUpon investigation, Okta determined that a new feature, “Schedule User Imports”, which allows imports to be scheduled for specific times was enabled for customers as part of the 2020.02.0 Monthly Production release*.* As part of release, existing scheduled interval imports were ported to the new day and time based feature which inadvertently caused a significant number of jobs to be run at a higher frequency. The increased volume of jobs resulted in increased wait times on some servers.\r\n\r\nAt approximately ‪2:45pm PST, Okta began to roll back the feature and terminate non-admin initiated jobs that were scheduled to execute on the impacted job servers. Jobs that were terminated would get triggered again on the next scheduled import. By ‪3:15pm PST, normal service was restored.\r\n\r\nOkta is re-evaluating design approaches to provide enhanced import scheduling capability, while ensuring reliability and performance at scale. Okta is also investigating enhancements to the alerting mechanism to improve diagnosis for this type of alert.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKsjUAE"},"Id":"a1P1Y000007zKsjUAE","CreatedDate":"2020-04-15T23:41:04.000+0000","IncidentId__c":"a9C1Y0000004EI1UAM","UpdateLog__c":"Okta has observed an issue with the new 5.0.2 version of Okta Verify for iOS. Some users cannot verify their identity by acknowledging the push notification sent to their iPhone. They can work around this issue by entering the verification code displayed on Okta Verify. We are working to identify the cause of the issue in order to implement a solution and will provide ongoing progress updates.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKsoUAE"},"Id":"a1P1Y000007zKsoUAE","CreatedDate":"2020-04-16T00:41:22.000+0000","IncidentId__c":"a9C1Y0000004EI1UAM","UpdateLog__c":"Okta has observed an issue with the new 5.0.2 version of Okta Verify for iOS. Some users cannot verify their identity by acknowledging the push notification sent to their iPhone. They can work around this issue by entering the verification code displayed on Okta Verify. We are continuing to work on identifying the cause of the issue in order to implement a solution and will provide ongoing progress updates. Please expect the next update at or before 7:30PM PDT.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKstUAE"},"Id":"a1P1Y000007zKstUAE","CreatedDate":"2020-04-16T02:48:34.000+0000","IncidentId__c":"a9C1Y0000004EI1UAM","UpdateLog__c":"Okta has observed an issue with the new 5.0.2 version of Okta Verify for iOS. Some users cannot verify their identity by acknowledging the push notification sent to their iPhone. They can work around this issue by entering the verification code displayed on Okta Verify. We are continuing to focus our efforts on identifying and testing a solution. Next update in approximately 2 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKsyUAE"},"Id":"a1P1Y000007zKsyUAE","CreatedDate":"2020-04-16T05:03:15.000+0000","IncidentId__c":"a9C1Y0000004EI1UAM","UpdateLog__c":"Okta has observed an issue with the new 5.0.2 version of Okta Verify for iOS. Some users cannot verify their identity by acknowledging the push notification sent to their iPhone. They can work around this issue by entering the verification code displayed on Okta Verify. We have identified the issue and are actively working on a fix. Next update in approximately 2 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKt3UAE"},"Id":"a1P1Y000007zKt3UAE","CreatedDate":"2020-04-16T07:49:07.000+0000","IncidentId__c":"a9C1Y0000004EI1UAM","UpdateLog__c":"Okta has observed an issue with the new 5.0.2 version of Okta Verify for iOS. Some users cannot verify their identity by acknowledging the push notification sent to their iPhone. Users should work around this issue by entering the verification code displayed on Okta Verify.\r\n\r\nWe have identified the issue and are actively working on a fix. As soon as a fix is available and thoroughly validated, Okta will submit it to the App Store. We expect the next update to be Thursday, April 16th at or before 9AM PDT.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKtIUAU"},"Id":"a1P1Y000007zKtIUAU","CreatedDate":"2020-04-16T15:15:00.000+0000","IncidentId__c":"a9C1Y0000004EI1UAM","UpdateLog__c":"Okta has observed an issue with the new 5.0.2 version of Okta Verify for iOS. Some users cannot verify their identity by acknowledging the push notification sent to their iPhone. Users should work around this issue by entering the verification code displayed on Okta Verify. \r\n\r\nWe have identified the issue and have a fix undergoing approval in the App Store. We expect the next update to be at or before 12PM PDT.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKtNUAU"},"Id":"a1P1Y000007zKtNUAU","CreatedDate":"2020-04-16T18:59:08.000+0000","IncidentId__c":"a9C1Y0000004EI1UAM","UpdateLog__c":"Okta has released a fix to Okta Verify (OV version 5.0.5) for users who updated to 5.0.2 and are experiencing push notifications issues.\r\n\r\nOkta admins may guide users currently experiencing an issue with push notifications to manually upgrade OV through the App Store to version 5.0.5. Okta admins should discourage users for whom push notifications are currently working, from upgrading to version 5.0.5, as this may cause their push notifications to break.\r\n\r\nOkta is actively working on an additional fix that will fully resolve the issue for all upgrade paths. This final version will be rolled out to all end users through a regular App Store app rollout.\r\n\r\nUsers experiencing push notifications issues may also continue to work around the push failure by entering the verification code displayed in Okta Verify.\r\n\r\nA root cause analysis will be posted here on April 21, 2020.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKuVUAU"},"Id":"a1P1Y000007zKuVUAU","CreatedDate":"2020-04-22T00:19:37.000+0000","IncidentId__c":"a9C1Y0000004EI1UAM","UpdateLog__c":"Root Cause Analysis: On April 15th at approximately 2pm PDT, Okta began receiving reports that some users could not complete push-based Okta Verify MFA flows on iOS devices.\r\n\r\nOkta determined that this issue was unique to version 5.0.2 of Okta Verify for iOS which had begun a phased rollout earlier that same day. At 2:20pm PDT Okta suspended the rollout of Okta Verify 5.0.2 for iOS and continued to investigate the problem. Customers were advised to work around the issue by entering the verification code displayed on Okta Verify.\r\n\r\nAfter further investigation, Okta determined that push verify on Okta Verify 5.0.2 was unable to complete verification for users who had last enrolled with Okta Verify on their device prior to Okta Verify version 4.4. This was due to an incorrect handling of legacy encryption keys that were used in previous Okta Verify versions.\r\n\r\nBecause of the complexity of the issue, Okta expedited a temporary build that would rollback the functionality for users affected as the full fix was worked on to support all upgrade paths. The build was approved and published at 11:00am PST on April 16th.\r\n\r\nOkta worked on a comprehensive fix, leveraging heuristics to determine the encryption method of the stored keys in order to resolve the issue for all users. This fix was submitted to the app store and expedited as Okta Verify 5.0.7 at 5:20pm PDT on April 16th, the build was approved and published at 9:20am PDT on April 17th.\r\n\r\nOkta is expanding migration testing scenarios across more versions to prevent future occurrences as well as incorporating additional monitoring to detect similar issues more quickly in the future.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QQzTUAW"},"Id":"a1P1Y000009QQzTUAW","CreatedDate":"2020-08-19T21:19:27.000+0000","IncidentId__c":"a9C1Y0000004EW8UAM","UpdateLog__c":"Root Cause Analysis: On August 13 at approximately 1:02pm PDT, Okta detected 502 bad gateway errors for requests in US Preview Cell 1.\r\n\r\nUpon investigation, Okta determined that during the deployment of an internal infrastructure change, modifications were introduced which resulted in virtual firewall security group rules being revoked. Rules revocation further restricted inbound traffic to hosts and caused all traffic to be blocked at the edge of the infrastructure.\r\n\r\nAt approximately 1:11pm PDT, the security group rules were manually reapplied, and service was fully restored.\r\n\r\nTo prevent recurrence of this issue, Okta has suspended the rollout of internal services until we ensure that it can be done with no customer impact.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QQziUAG"},"Id":"a1P1Y000009QQziUAG","CreatedDate":"2020-08-20T22:59:21.000+0000","IncidentId__c":"a9C1Y0000008PYSUA2","UpdateLog__c":"An issue impacting authentication, API services and multi factor authentication in US Cells 1 - 4 is currently being investigated.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QQznUAG"},"Id":"a1P1Y000009QQznUAG","CreatedDate":"2020-08-20T23:32:13.000+0000","IncidentId__c":"a9C1Y0000008PYSUA2","UpdateLog__c":"Services are starting to return in US Cells 1 - 4. Okta is actively working to fully restore all services as quickly as possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QQzsUAG"},"Id":"a1P1Y000009QQzsUAG","CreatedDate":"2020-08-21T00:04:07.000+0000","IncidentId__c":"a9C1Y0000008PYSUA2","UpdateLog__c":"Resolved: Okta has successfully restored all services in US Cell 1 - 4. Okta will continue to monitor the situation carefully. Additional root cause information will be provided within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QQzxUAG"},"Id":"a1P1Y000009QQzxUAG","CreatedDate":"2020-08-23T17:50:08.000+0000","IncidentId__c":"a9C1Y0000008PYSUA2","UpdateLog__c":"On August 20 at approximately 3:29pm PDT, Okta began to receive alerts for HTTP 503 Service Unavailable errors for US Cells 1, 2, 3 & 4. The impact varied by cell and features in use.\r\n\r\nImpact across US Cells 1, 2, 3 and 4:\r\n-Users were unable to authenticate into orgs with Custom Domains until 4:00PM PDT\r\n-AD and LDAP agents could not connect to Okta until 4:17PM PDT. New Delegated Authentication users (who had not previously authenticated), AD and LDAP imports and AD password sync functionality were all impacted until the agents connected\r\n-The System Log and the logs API may have experienced high latency or failed with 5xx responses\r\n\r\nUS Cell 1 specific additional impact:\r\n-No impact to users beyond what is described for all cells\r\n-Access to the System Log and the logs API was fully restored at 4:02PM PDT\r\n\r\nUS Cell 2 specific additional impact:\r\n-Unavailable for 20 minutes until 3:49pm\r\n-Access to the System Log and the logs API was fully restored at 3:46PM PDT\r\n\r\nUS Cell 3 specific additional impact:\r\n-Unavailable for 14 minutes until 3:34pm\r\n-Access to the System Log and the logs API was fully restored at 4:28PM PDT\r\n\r\nUS Cell 4 specific additional impact:\r\n-Unavailable for 20 minutes until 3:49pm\r\n-Access to the System Log and the logs API was fully restored at 3:47PM PDT\r\n\r\nAt 3:15pm PDT, Okta started a manual maintenance operation to remove unused legacy virtual firewall rules for replaced infrastructure. At 3:29pm PDT, an incorrect command was executed which also deleted active firewall rules for current infrastructure in US Cells 1, 2, 3 & 4. As Okta maintains a restrictive firewall policy that defaults to denying access unless expressly allowed by a rule, rules revocation further restricted inbound traffic to hosts and caused traffic to be blocked. Okta was immediately alerted by our own monitoring tools and at 3:41pm PDT, Okta began to restore service by reapplying firewall rules in the affected cells. Full service across all cells was restored by 4:28PM PDT and for the different services as noted above. \r\n\r\nWe sincerely apologize for the impact to affected customers and are working to improve the Okta service. To prevent recurrence, Okta has implemented a moratorium on maintenance infrastructure changes until a full review of the legacy infrastructure has been completed, and additional manual process checks and improvements in automation are incorporated to ensure modification to security group changes are done without customer impact.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QR1UUAW"},"Id":"a1P1Y000009QR1UUAW","CreatedDate":"2020-08-26T18:28:53.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"We have identified the cause and have tested the fix.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QR1ZUAW"},"Id":"a1P1Y000009QR1ZUAW","CreatedDate":"2020-08-26T18:31:24.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"Deployment has begun in final test and we are confirming full resolution.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QR1eUAG"},"Id":"a1P1Y000009QR1eUAG","CreatedDate":"2020-08-26T20:40:02.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"Testing progressing well. Next scheduled update 2:00PM PDT","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QR1jUAG"},"Id":"a1P1Y000009QR1jUAG","CreatedDate":"2020-08-26T21:03:07.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"Test runs complete proceding to next stage. Additional update targeted for 3:00pm PDT.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QR1oUAG"},"Id":"a1P1Y000009QR1oUAG","CreatedDate":"2020-08-26T21:52:06.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"Deployment and testing has confirmed resolution on US Cell 7.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QR1tUAG"},"Id":"a1P1Y000009QR1tUAG","CreatedDate":"2020-08-26T21:52:33.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"Proceding with further production deployment.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QR1yUAG"},"Id":"a1P1Y000009QR1yUAG","CreatedDate":"2020-08-26T23:11:29.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"We have customer confirmation of the fix on US Cell 7, and continue deploying it to the remaining cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QR23UAG"},"Id":"a1P1Y000009QR23UAG","CreatedDate":"2020-08-26T23:59:48.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"We now have customer confirmation of the fix on US Cell 3, and US Cell 7. Deployment continues to the remaining cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QR28UAG"},"Id":"a1P1Y000009QR28UAG","CreatedDate":"2020-08-27T01:28:03.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"Resolved: An issue impacting the Helpdesk Admin Role on US Cell 1, US Cell 2, US Cell 3, US Cell 5, US Cell 7, US Cell 11, and EMEA Cell 1 has been resolved. The issue could prevent Helpdesk Admins from being able to reset a user’s password via a temporary password. A fix has been deployed and we have confirmed that the issue has been resolved. Additional root cause information will be provided within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QR31UAG"},"Id":"a1P1Y000009QR31UAG","CreatedDate":"2020-08-28T21:43:58.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"Root Cause Analysis: On Wednesday August 26th at approximately 7:40 AM PDT, Okta began receiving customer reports that Help desk Admins were unable to trigger a user password reset through the Okta Admin UI.\r\n\r\nUpon investigation, it was determined that Weekly Release 2020.08.2, which included a fix to remove permissions allowing Help desk Admins to expire passwords for all Okta-sourced users in the org, introduced a regression. Due to this regression, Help desk Admins were unable to trigger a user password reset through the Okta Admin UI. Okta Preview Cells OP1, OP2, and OP3 were impacted, as well as Production Cells US 2, 3, 5, 7, 11 and EU1 were impacted.\r\n\r\nThroughout the incident, password reset functionality remained available for Super, Org, and Group Admins; Reset password functionality for end users also continued to function normally.\r\n\r\nAt approximately 2:43 PM PDT Okta engineering deployed a fix to US Cell 7, which reverted the regression causing change. This fix was confirmed to restore password reset functionality for Help desk Admins and was subsequently rolled out to all impacted cells (Preview Cells OP1, OP2 and OP3, US Cells 2, 3, 5, 11, and EU1). At 5:23 PM PDT Okta engineering confirmed the fix had been deployed and validated across all impacted cells.\r\n\r\nOkta is implementing additional testing to protect against future regressions related to changes in role-based permission settings. The change that caused the incident will be rolled out again only after additional validation and safeguards are put in place.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRAMUA4"},"Id":"a1P1Y000009QRAMUA4","CreatedDate":"2020-09-28T23:02:42.000+0000","IncidentId__c":"a9C1Y0000008Pa9UAE","UpdateLog__c":"Okta is currently experiencing an issue impacting Microsoft applications in all cells where SSO & provisioning are experiencing failures. Okta has identified this issue is related to the current Microsoft Azure outage impacting many Microsoft customers. Further information is available from Microsoft via https://status.azure.com/en-us/status","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRARUA4"},"Id":"a1P1Y000009QRARUA4","CreatedDate":"2020-09-29T00:28:33.000+0000","IncidentId__c":"a9C1Y0000008Pa9UAE","UpdateLog__c":"Single-Sign On services related to Microsoft O365 seem to be returning to normal. There are still issues with provisioning that Okta is monitoring for improvement. Microsoft has taken mitigation steps and further information is available from Microsoft via https://status.azure.com/en-us/status","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRAWUA4"},"Id":"a1P1Y000009QRAWUA4","CreatedDate":"2020-09-29T02:08:31.000+0000","IncidentId__c":"a9C1Y0000008Pa9UAE","UpdateLog__c":"O365 Provisioning is starting to recover. Okta is continuing to monitor all cells for Provisioning issues. Microsoft is continuing to work towards full recovery from this outage and further information is available from Microsoft via https://status.azure.com/en-us/status","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRAbUAO"},"Id":"a1P1Y000009QRAbUAO","CreatedDate":"2020-09-29T05:02:13.000+0000","IncidentId__c":"a9C1Y0000008Pa9UAE","UpdateLog__c":"O365 Provisioning is starting to recover but Okta is still seeing intermittent provisioning failures. Okta is continuing to monitor and working closely with Microsoft to fully recover O365 provisioning in all cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRAgUAO"},"Id":"a1P1Y000009QRAgUAO","CreatedDate":"2020-09-29T12:57:25.000+0000","IncidentId__c":"a9C1Y0000008Pa9UAE","UpdateLog__c":"RESOLVED: Issues impacting O365 Single Sign-On and 0365 Provisioning have been resolved. Okta and Microsoft have resolved the remaining intermittent provisioning issues. Okta will continue to monitor the situation carefully. For customers using AAD graph API, residual errors may be observed when Okta tries to connect to Microsoft. To resolve these issues, administrators can go to provisioning tab for the application instances, click edit, test credential and save the app instances to resolve the issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRAlUAO"},"Id":"a1P1Y000009QRAlUAO","CreatedDate":"2020-09-30T15:48:01.000+0000","IncidentId__c":"a9C1Y0000008Pa9UAE","UpdateLog__c":"Root Cause Analysis:\r\nOn September 28 at approximately 2:36pm PDT, Okta detected that customers across all Okta cells were experiencing SSO and provisioning failures to Microsoft O365 services.\r\n \r\nUpon investigation, it was determined that this issue was related to a disruption in Microsoft’s Azure AD services. Per Microsoft, \"Users who were not already authenticated to the cloud services using Azure AD were primarily impacted.\" Additional information including details on incident root cause can be found at https://status.office.com/ and https://status.azure.com/en-us/status/history/\r\n \r\nAt approximately 5:02pm PDT, O365 Single Sign-On and O365 Provisioning began to recover with intermittent failures still being reported. On September 29 at approximately 2:40 am PDT, service was fully restored.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRBjUAO"},"Id":"a1P1Y000009QRBjUAO","CreatedDate":"2020-10-27T22:54:59.000+0000","IncidentId__c":"a9C1Y0000008Pd8UAE","UpdateLog__c":"Service is fully restored for Advanced Server Access. Additional root cause information will be provided within 48 business hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRC3UAO"},"Id":"a1P1Y000009QRC3UAO","CreatedDate":"2020-10-30T04:45:44.000+0000","IncidentId__c":"a9C1Y0000008Pd8UAE","UpdateLog__c":"Root Cause Analysis:\r\nOn October 27 at 2:56PM (PT) Okta detected an increase in HTTP 500 Internal Server Errors, impacting the ASA service. Customers attempting to use the ASA client or the dashboard, as well as all server agents attempting to enroll or sync data, saw failed requests.\r\n\r\nAfter investigation, it was determined that the root cause was a migration to new AWS Auto Scaling Groups (ASG). During this migration both the old ASGs and the new ASGs were active, which had the effect of doubling the maximum number of connections which could be opened to the primary application database cluster. An unexpected significant spike in API traffic mid-migration pushed the number of connections to this maximum. The database cluster was not configured to support this number of connections, causing the database cluster to become unresponsive.\r\n\r\nAt approximately 3:10PM (PT) Okta engineers reverted the migration, directing all traffic to the original ASGs, and began downsizing the new ASGs to reduce the number of open connections. Engineers simultaneously triggered a failover to the secondary DB cluster and restarted the primary DB cluster. Unfortunately, the ASGs had still not fully downsized, and all ASG instances connected to the secondary DB cluster caused similar issues on the secondary DB cluster. At this point, engineers disabled serving agent API requests in an attempt to reduce load. Engineers then manually forced termination of the remaining nodes in the ASGs, and restarted both DB clusters. At 3:39PM (PT) Okta validated that server login flows were being serviced normally. After establishing that the service was stable, Okta engineers re-enabled the serving of agent API requests, fully restoring all service functionality at 3:46PM (PT).\r\n\r\nOkta has implemented additional monitoring of ASA’s database performance for early detection, and engineers are in the process of reviewing and tuning ASA’s database connection limits. Additionally, Okta plans to implement a connection-pooling proxy in front of the primary ASA database cluster to prevent future occurrences of this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRDLUA4"},"Id":"a1P1Y000009QRDLUA4","CreatedDate":"2020-11-25T20:12:41.000+0000","IncidentId__c":"a9C1Y0000008PepUAE","UpdateLog__c":"We have identified the root cause and are investigating mitigations to address the root cause. The processing of System Log, Search API, webhooks and Workflows Events associated with the Okta connector remain impacted in US Cells 1, 2, 3, 4, and US Preview 1. Next update: 2PM PST.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRDQUA4"},"Id":"a1P1Y000009QRDQUA4","CreatedDate":"2020-11-25T22:11:36.000+0000","IncidentId__c":"a9C1Y0000008PepUAE","UpdateLog__c":"We have identified the root cause and are continuing to investigate mitigations to address the root cause. The processing of System Log, Search API, webhooks and Workflows Events associated with the Okta connector remain impacted in US Cells 1, 2, 3, 4, and US Preview 1. We will provide the next update by 5PM PST.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z0000051aauUAA"},"Id":"a1P0Z0000051aauUAA","CreatedDate":"2018-05-31T08:06:30.000+0000","IncidentId__c":"a9C0Z0000004E5UUAU","UpdateLog__c":"An issue impacting all functionality in US Cell 6 is being investigated. It appears that the underlying infrastructure issue has been resolved, and we are monitoring closely to ensure service has been restored.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XZAoUAO"},"Id":"a1P0Z000005XZAoUAO","CreatedDate":"2018-09-04T14:37:01.000+0000","IncidentId__c":"a9C0Z0000008PTrUAM","UpdateLog__c":"An issue causing intermittent authentication issues to Microsoft Office 365 for a subset of end-users and administrators in all Cells is being investigated. Microsoft has indicated they are experiencing a data center issue resulting in a degraded Office 365 user experience and are actively working to resolve the issue. Further information is available from Microsoft via [https://status.office365.com](https://status.office365.com).","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XZBwUAO"},"Id":"a1P0Z000005XZBwUAO","CreatedDate":"2018-09-04T22:43:47.000+0000","IncidentId__c":"a9C0Z0000008PTwUAM","UpdateLog__c":"Okta is working with Google to remove the \"deceptive site\" flagging for *.oktapreview.com from the Google Safe Browsing list. We are working to resolve the issue as quickly as possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XZCGUA4"},"Id":"a1P0Z000005XZCGUA4","CreatedDate":"2018-09-05T03:16:40.000+0000","IncidentId__c":"a9C0Z0000008PTwUAM","UpdateLog__c":"Resolved: An issue impacting access to *.oktapreview.com due to a \"Deceptive site\" flagging from the Google Safe Browsing list on Google Chrome and Firefox has been resolved. During this event, End-Users and Admins may have encountered a \"Deceptive site ahead\" warning while attempting to login to their Okta Preview tenant. Additional root cause information will provided within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XZCLUA4"},"Id":"a1P0Z000005XZCLUA4","CreatedDate":"2018-09-05T03:53:22.000+0000","IncidentId__c":"a9C0Z0000008PTrUAM","UpdateLog__c":"Resolved: An issue impacting authentication and system responsiveness for a subset of end-users and administrators accessing Microsoft Office 365 has subsided. Further investigation of the issue found the errors were correlated with a service degradation reported by Microsoft. Further information is available from Microsoft via https://status.office365.com.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XZDEUA4"},"Id":"a1P0Z000005XZDEUA4","CreatedDate":"2018-09-05T18:42:35.000+0000","IncidentId__c":"a9C0Z0000008PU1UAM","UpdateLog__c":"An issue causing intermittent authentication to Microsoft Office 365 for a subset of end-users and administrators in all Cells is being investigated. Users attempting to authenticate with Office 365 are receiving a \"Throttled\" message. We are actively working with Microsoft who has reported this is related to mitigation activities from yesterday's service disruption (MO147606):\r\n\r\n\"As part of our follow-up remediation actions stemming from the issue reported under service incident MO147606, an update was introduced to the components that manage authentication. We’ve determined that this update has resulted in users receiving a message indicating they are being throttled when attempting to access Outlook and Skype. We’re reverting the update to remediate the problem”","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XZDOUA4"},"Id":"a1P0Z000005XZDOUA4","CreatedDate":"2018-09-05T19:16:27.000+0000","IncidentId__c":"a9C0Z0000008PU1UAM","UpdateLog__c":"We are actively working with Microsoft who has reported this is related to mitigation activities from yesterday's service disruption (MO147606):\r\n\r\n\r\n\"We identified and reverted an update that was causing multiple services to be throttled. We're conducting tests to ensure that impact has been resolved. Further updates can be found in the admin portal under EX147785 and MO147789.\"\r\n\r\nOkta will post additional information as we learn it from Microsoft.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XZDTUA4"},"Id":"a1P0Z000005XZDTUA4","CreatedDate":"2018-09-05T19:31:37.000+0000","IncidentId__c":"a9C0Z0000008PU1UAM","UpdateLog__c":"Resolved: An issue impacting authentication and system responsiveness for a subset of end-users and administrators accessing Microsoft Office 365 has subsided. Further investigation of the issue found the errors were correlated with a service degradation reported by Microsoft. Microsoft reported this is related to mitigation activities form yesterday's service disruption (MO147606):\r\n\r\n\"We identified and reverted an update that was causing multiple services to be throttled. We're conducting tests to ensure that impact has been resolved. Further updates can be found in the admin portal under EX147785 and MO147789.\"","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XZU5UAO"},"Id":"a1P0Z000005XZU5UAO","CreatedDate":"2018-09-18T01:23:42.000+0000","IncidentId__c":"a9C0Z0000008PUVUA2","UpdateLog__c":"End users are currently experiencing errors when authenticating to O365. We are actively investigating and will update this message with more information as soon as we have it.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XZUAUA4"},"Id":"a1P0Z000005XZUAUA4","CreatedDate":"2018-09-18T01:55:55.000+0000","IncidentId__c":"a9C0Z0000008PUVUA2","UpdateLog__c":"End users are currently experiencing intermittent errors when authenticating to O365. Okta is executing a procedure to resolve the errors with most clients now encountering no errors. We will update this message with more information as soon as we have it.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XaKaUAK"},"Id":"a1P0Z000005XaKaUAK","CreatedDate":"2018-10-15T23:35:51.000+0000","IncidentId__c":"a9C0Z0000008PVOUA2","UpdateLog__c":"An issue impacting authentication and system responsiveness for a subset of end-users and administrators in US Cell 6 is actively being mitigated. System responsiveness and major functionality have returned to normal. We are continuing to investigate the root cause and will continue to provide updates until all issues are fully resolved.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XaKfUAK"},"Id":"a1P0Z000005XaKfUAK","CreatedDate":"2018-10-16T00:08:32.000+0000","IncidentId__c":"a9C0Z0000008PVOUA2","UpdateLog__c":"An issue impacting authentication and system responsiveness for a subset of end-users and administrators in US Cell 6 has been resolved. System responsiveness and major functionality have been restored l. We are currently investigating the root cause and will post the full Root Cause Analysis within the next 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XaP6UAK"},"Id":"a1P0Z000005XaP6UAK","CreatedDate":"2018-10-18T04:22:28.000+0000","IncidentId__c":"a9C0Z0000008PVOUA2","UpdateLog__c":"Resolved: On October 15th, beginning at 3:58 PM PDT, Okta experienced a service disruption in US Cell 6. The service disruption occurred during two distinct windows. Between 3:58 PM PDT and 4:05 PM PDT, approximately 15% of incoming end user requests to US Cell 6 experienced latency or errors. Between 4:28 PM PDT and 4:33 PM PDT, approximately 9% of end-user requests to US Cell 6 experienced latency or errors. The service disruption was resolved and the service was returned to normal at 4:33pm PDT.\r\n\r\nThe root cause of the service disruptions was traced to a significant and abnormal increase in requests that activated Okta's rate limiting framework but eventually resulted in higher than normal CPU utilization within our infrastructure. Starting at 3:57, Okta's monitoring services detected the increase in abnormal traffic, and automated service protections were employed to mitigate the impact, however, additional actions were needed to reroute and isolate the offending traffic.\r\n\r\nFollowing the service disruption, Okta reviewed the incident response and has identified and plans to implement additional automated and manual incident response process improvements and infrastructure changes to further minimize impact or prevent similar incidents in the future.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XaYwUAK"},"Id":"a1P0Z000005XaYwUAK","CreatedDate":"2018-10-25T03:43:48.000+0000","IncidentId__c":"a9C0Z0000008PViUAM","UpdateLog__c":"Resolved: An issue impacting compatibility with Office 365 user provisioning has been resolved. Root cause analysis will be posted here within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005XasDUAS"},"Id":"a1P0Z000005XasDUAS","CreatedDate":"2018-11-12T00:10:58.000+0000","IncidentId__c":"a9C0Z0000008PW7UAM","UpdateLog__c":"We continue to observe delays or failures in the delivery of outbound SMS to the AT&T network in the US. Most end users should be able to succeed with a retry. SMS sent to other networks are unaffected. We are actively working with our providers to resolve the issue and will provide an update when we have further information. Okta recommends affected AT&T mobile customers use an alternate second factor as a work around to the issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000005Xb1FUAS"},"Id":"a1P0Z000005Xb1FUAS","CreatedDate":"2018-11-14T19:13:55.000+0000","IncidentId__c":"a9C0Z0000008PW7UAM","UpdateLog__c":"Resolved: Okta received reports of delays and failures in the delivery of SMS for multi-factor authentication. Upon further investigation, it was determined that the issue was caused by an on-going AT&T cellular network incident. Okta worked with its service providers to raise the issue and worked with customers to workaround the the issue until service was fully restored by AT&T. Users who were not on the AT&T cellular network or that leveraged other authentication factors were not affected throughout the incident.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eJNEUA2"},"Id":"a1P0Z000007eJNEUA2","CreatedDate":"2018-04-23T16:41:04.000+0000","IncidentId__c":"a9C0Z000000TNelUAG","UpdateLog__c":"Okta is currently investigating an issue with the /api/v1/events endpoint in all cells. To access system logs, please utilize the [System Log API Endpoint](https://developer.okta.com/docs/api/resources/system_log).","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eJO7UAM"},"Id":"a1P0Z000007eJO7UAM","CreatedDate":"2018-04-23T17:13:24.000+0000","IncidentId__c":"a9C0Z000000TNelUAG","UpdateLog__c":"Okta is currently investigating an issue with the /api/v1/events endpoint in all cells. To access system logs, please utilize the [System Log API Endpoint](https://developer.okta.com/docs/api/resources/system_log).","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eJOWUA2"},"Id":"a1P0Z000007eJOWUA2","CreatedDate":"2018-04-23T17:44:08.000+0000","IncidentId__c":"a9C0Z000000TNelUAG","UpdateLog__c":"Okta continues to investigate an issue with the /api/v1/events endpoint in all cells. To access system logs, please utilize the [System Log API Endpoint](https://developer.okta.com/docs/api/resources/system_log). The next update will take place at 11:45am PT or as soon as additional details become available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eJPPUA2"},"Id":"a1P0Z000007eJPPUA2","CreatedDate":"2018-04-23T18:43:56.000+0000","IncidentId__c":"a9C0Z000000TNelUAG","UpdateLog__c":"Okta continues to investigate an issue with the /api/v1/events endpoint in all cells. To access system logs, please utilize the [System Log API Endpoint](https://developer.okta.com/docs/api/resources/system_log). The next update will take place at 12:45pm PT or as soon as additional details become available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eJPoUAM"},"Id":"a1P0Z000007eJPoUAM","CreatedDate":"2018-04-23T19:06:35.000+0000","IncidentId__c":"a9C0Z000000TNelUAG","UpdateLog__c":"Okta is actively mitigating an issue with the /api/v1/events endpoint in all cells. To access system logs, please utilize the [System Log API Endpoint](https://developer.okta.com/docs/api/resources/system_log). The next update will take place at 1:00pm PT or as soon as additional details become available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eJQ8UAM"},"Id":"a1P0Z000007eJQ8UAM","CreatedDate":"2018-04-23T19:54:35.000+0000","IncidentId__c":"a9C0Z000000TNelUAG","UpdateLog__c":"Okta continues to mitigate an issue with the /api/v1/events endpoint in all cells. To access system logs, please utilize the [System Log API Endpoint](https://developer.okta.com/docs/api/resources/system_log). The next update will take place at 2:00pm PT or as soon as additional details become available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eJQhUAM"},"Id":"a1P0Z000007eJQhUAM","CreatedDate":"2018-04-23T21:01:13.000+0000","IncidentId__c":"a9C0Z000000TNelUAG","UpdateLog__c":"Okta continues to mitigate an issue with the /api/v1/events endpoint in all cells. To access system logs, please utilize the [System Log API Endpoint](https://developer.okta.com/docs/api/resources/system_log). The next update will take place at 3:00pm PT or as soon as additional details become available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eJUoUAM"},"Id":"a1P0Z000007eJUoUAM","CreatedDate":"2018-04-24T15:17:52.000+0000","IncidentId__c":"a9C0Z000000TNeqUAG","UpdateLog__c":"An issue impacting authentication for a subset of end users in US Cell 2 is being investigated.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eJVNUA2"},"Id":"a1P0Z000007eJVNUA2","CreatedDate":"2018-04-24T15:55:38.000+0000","IncidentId__c":"a9C0Z000000TNeqUAG","UpdateLog__c":"An issue impacting authentication for a subset of end users in US Cell 2 was addressed. Additional root cause information will provided when available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eJbWUAU"},"Id":"a1P0Z000007eJbWUAU","CreatedDate":"2018-04-25T02:43:56.000+0000","IncidentId__c":"a9C0Z000000TNeqUAG","UpdateLog__c":"Root Cause Analysis:\r\n\r\nOn April 24th, at 7:53am PDT, Okta experienced a broad service disruption to US Cell 2. End users in US Cell 2 attempting to reach Okta received HTTP 500 errors, and customer integrations attempting to reach Okta API endpoints in US Cell 2 encountered elevated response times and errors. Operational mitigation steps commenced immediately at 7:53am. The greatest frequency of errors was a 14 minute period between 7:53am and 8:07am. The success rate began to gradually improve from 8:07am, and by 8:25am approximately 80% of requests were successfully returned. The cell was fully restored by 8:48am. \r\n\r\nRoot cause was traced to a combination of a procedural error and an automation bug while security patching infrastructure within US Cell 2. The procedural error targeted the security patching to a high usage cache cluster. The automation used to apply the patch disrupted the cluster’s ability to failover across nodes. \r\nTo prevent recurrence of this type of issue in the future, Engineering has immediately implemented additional safeguards to review and confirm operational patching procedures. Additionally, Engineering has prioritized a deep technical assessment to strengthen the patching automation for cache clusters.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eNdTUAU"},"Id":"a1P0Z000007eNdTUAU","CreatedDate":"2018-05-17T19:24:28.000+0000","IncidentId__c":"a9C0Z000000TNgDUAW","UpdateLog__c":"Okta is actively working to mitigate the issues impacting authentication and user interactive errors in US Cell 6. As of 12:19pm PDT, error rates have returned to 0%. We are continuing to monitor and will post updates as we continue to address the root cause.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eNdsUAE"},"Id":"a1P0Z000007eNdsUAE","CreatedDate":"2018-05-17T19:46:07.000+0000","IncidentId__c":"a9C0Z000000TNgDUAW","UpdateLog__c":"An issue impacting authentication was resolved at 12:19pm PDT. During the event, End-Users and Administrators may have experienced HTTP 5xx errors while attempting to authenticate to their Okta tenant. Additional root cause information will be provided within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P0Z000007eOBkUAM"},"Id":"a1P0Z000007eOBkUAM","CreatedDate":"2018-05-20T09:27:36.000+0000","IncidentId__c":"a9C0Z000000TNgDUAW","UpdateLog__c":"On May 17th, beginning at 11:28am PDT, Okta experienced a service disruption in US Cell 6. End-User and Administrators attempting to authenticate to their Okta tenant may have experienced HTTP 5xx errors. Through the duration of the incident, approximately 30% of US Cell 6 users attempting primary or multi-factor authentication encountered errors and would have needed to retry. Other requests, and End-Users and Administrators who already had an active session did not experience any impact.\r\n<br><br>\r\nThe issue surfaced when the cell's session token management cluster became overwhelmed by a sudden increase in the number of active sessions. Automated monitoring failed to effectively alert before the cluster was nearing its maximum capacity.\r\n<br><br>\r\nThe issue was resolved by 12:19pm PDT after operational actions were taken to manage the session count while capacity was added to the cluster. Immediately following the incident, Okta corrected the monitoring gap to prevent a future recurrence. Architectural design work is being initiated to better manage session counts and their expiry.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V3R2UAK"},"Id":"a1P1Y000005V3R2UAK","CreatedDate":"2018-12-13T13:40:43.000+0000","IncidentId__c":"a9C1Y000000GpJuUAK","UpdateLog__c":"An issue impacting CORS authentication in Okta Preview Cells has been identified and customers using OIDC applications and API AM might be impacted. A fix has been identified and we are working to return the service to normal as quickly as possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V3R7UAK"},"Id":"a1P1Y000005V3R7UAK","CreatedDate":"2018-12-13T14:15:49.000+0000","IncidentId__c":"a9C1Y000000GpJuUAK","UpdateLog__c":"An issue impacting CORS authentication in Okta Preview Cells has been identified and customers using OIDC applications and API AM might be impacted. A fix has been identified and we are working to return the service to normal as quickly as possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V3RCUA0"},"Id":"a1P1Y000005V3RCUA0","CreatedDate":"2018-12-13T14:51:42.000+0000","IncidentId__c":"a9C1Y000000GpJuUAK","UpdateLog__c":"Resolved: The issue impacting CORS authentication for customers using OIDC applications and API AM in Okta Preview Cells has been fixed. Service has been restored to normal.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V3RHUA0"},"Id":"a1P1Y000005V3RHUA0","CreatedDate":"2018-12-17T07:38:38.000+0000","IncidentId__c":"a9C1Y000000GpJuUAK","UpdateLog__c":"Root Cause Analysis: On December 12th, an issue was introduced in Okta Preview cells which impacted the use of the Auth SDKs. Specifically, customers using version 2.0 or higher of Okta’s JavaScript SDK AuthJS (@okta/okta-auth-js) to leverage OAuth 2.0 may have experienced authentication failures starting at approximately 2:21pm PST. Additionally, customers using Okta’s Angular, React and Vue integration libraries and OAuth 2.0 in Preview cells were also impacted by this issue. The degree of impact varied depending on each customers’ implementation and client-side caching. \r\n<br><br>\r\nThe root cause was traced to an unexpected side effect from a fix applied to our CORS/TrustedOrigins framework. The issue was fully resolved on Dec 13th by 6:26am PST in US Preview Cell 1 and by 6:38am PST in EU Preview Cell 2. To prevent a similar re-occurrence, the test automation framework gap has been updated and existing test suites are being enhanced to identify such errors in the future..","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V3qHUAS"},"Id":"a1P1Y000005V3qHUAS","CreatedDate":"2019-02-06T01:10:05.000+0000","IncidentId__c":"a9C1Y000000GpKsUAK","UpdateLog__c":"Okta is currently experiencing an issue in EU1, OP1, OK1, OK2, OK3, and OK5. Some office 365 users are seeing provisioning errors \"An error occurred while assigning this app\" when assigning users to the application.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V3qMUAS"},"Id":"a1P1Y000005V3qMUAS","CreatedDate":"2019-02-06T01:40:53.000+0000","IncidentId__c":"a9C1Y000000GpKsUAK","UpdateLog__c":"A fix has been identified where office 365 users are seeing provisioning errors. We are working to return the service to normal as quickly as possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V3qRUAS"},"Id":"a1P1Y000005V3qRUAS","CreatedDate":"2019-02-06T02:10:37.000+0000","IncidentId__c":"a9C1Y000000GpKsUAK","UpdateLog__c":"A fix has been identified where office 365 users are seeing provisioning errors. We are working to return the service to normal as quickly as possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V3scUAC"},"Id":"a1P1Y000005V3scUAC","CreatedDate":"2019-02-08T01:05:10.000+0000","IncidentId__c":"a9C1Y000000GpKxUAK","UpdateLog__c":"Okta is currently experiencing an issue with Office 365 provisioning. We are actively investigating and will update this message with more information as soon as we have it.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V3smUAC"},"Id":"a1P1Y000005V3smUAC","CreatedDate":"2019-02-08T01:34:33.000+0000","IncidentId__c":"a9C1Y000000GpKxUAK","UpdateLog__c":"A fix has been identified where office 365 users are seeing provisioning errors. We are working to return the service to normal as quickly as possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V3swUAC"},"Id":"a1P1Y000005V3swUAC","CreatedDate":"2019-02-08T10:37:44.000+0000","IncidentId__c":"a9C1Y000000GpKsUAK","UpdateLog__c":"On Feb 5th at approximately 4:10pm PDT, Okta began receiving sporadic reports from customers in US Cell 1 and EMEA Cell 1 who were experiencing errors while attempting to provision users to Office 365 or modifying Office365 configuration settings. User authentication was not impacted at any point.\r\nThe root cause was traced to a serialization bug that affected a portion of Office365 application tenants. In particular, those where federation was auto configured using the Okta Admin UI prior to 2016, or manually configured by running the Okta provided Azure AD power-shell cmdlets.\r\n\r\nOnce the root cause was identified, Okta deployed a fix starting at 5:17pm PDT on Feb 5. At 7:42pm PDT, the fix was completely rolled out to all production cells, and functionality was fully restored.\r\n\r\nOkta has updated its proactive monitoring to identify and prevent future occurrences.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005V49YUAS"},"Id":"a1P1Y000005V49YUAS","CreatedDate":"2019-02-19T10:25:14.000+0000","IncidentId__c":"a9C1Y000000GpKxUAK","UpdateLog__c":"On Feb 7th at approximately 4:30PM PDT, Okta began receiving reports from customers in US Cell 4 who were experiencing errors while attempting to provision users to Office 365 or while modifying Office 365 configuration settings. User authentication was not impacted at any point.\r\n<br><br>\r\nThe root cause was traced to an inability in some cases to read domain federation settings from Microsoft Office 365. This bug caused a failure to successfully provision users and save Office 365 app settings. Okta also determined that this issue was present in all cells, but only affected a subset of customers.\r\n<br><br>\r\nOnce the root cause was identified, Okta deployed a fix starting at 4:49pm PDT on Feb 7. at approximately 5:30pm PDT, the fix was successfully rolled out to all production cells, and functionality was fully restored.\r\n<br><br>\r\nAdditional monitoring to detect and mitigate this scenario has been rolled out for all environments. Testing will be augmented to prevent future occurrences.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qHdsUAE"},"Id":"a1P1Y000005qHdsUAE","CreatedDate":"2019-05-15T23:45:16.000+0000","IncidentId__c":"a9C1Y000000fxVtUAI","UpdateLog__c":"RESOLVED: Okta has identified and resolved an issue resulting in HTTP 400 errors when users were attempting to authenticate under certain CORS use-cases with tenants in US Preview Cell 1. Additional root cause information will provided within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qHo7UAE"},"Id":"a1P1Y000005qHo7UAE","CreatedDate":"2019-05-24T13:57:58.000+0000","IncidentId__c":"a9C1Y000000fxVtUAI","UpdateLog__c":"ROOT CAUSE: On May 15th at approximately 1:42PM PDT, customers in US Preview Cell 2 experienced elevated errors when making HTTP OPTIONS-based requests, predominately against the \"/oauth2/v1/userinfo\" and \"/api/v1/sessions/*****\" endpoints. At 3:09PM PDT, Okta began to detect similar errors in US Preview Cell 1.\r\n\r\nThe elevated error rate was determined to be due to a bug within a new framework used to determine request timeout configurations, and specifically impacted the HTTP OPTIONS method. The new timeout framework was disabled to mitigate the issue, and the service was fully restored at approximately 4:35PM PDT for US Preview Cell 1, and at 4:43PM PDT for US Preview Cell 2.\r\n\r\nFollowing this incident, this bug in the new timeout framework has been addressed. Okta will be adding additional test procedures to detect this issue prior to re-enabling the framework.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qHx9UAE"},"Id":"a1P1Y000005qHx9UAE","CreatedDate":"2019-06-05T01:23:26.000+0000","IncidentId__c":"a9C1Y000000fxWSUAY","UpdateLog__c":"Root Cause: On May 31st, at approximately 12:24 PM PDT Okta received reports from some customers receiving a '400 Bad Request' response while attempting to login with Agentless Desktop SSO using Chrome browsers.\r\n<br><br>\r\nUpon investigation, Okta determined that the issue impacted customers with either incorrect Chrome settings or network configuration (see https://help.okta.com/en/prod/Content/Topics/Directory/Configuring_Agentless_SSO.htm). Okta also determined that the issue had been exposed by the release of a software performance optimization which did not account for this possible browser mis-configuration; prior to the optimization, this mis-configuration would have resulted in users being redirected to the default login page.\r\n<br><br>\r\nGiven the impact and poor user experience, starting on May 31st at 2:30 PM PDT, Okta deployed a change that reverted behavior to the pre-optimization one. The deployment completed to all cells at approximately 5:30PM PDT.\r\n<br><br>\r\nOkta is implementing additional testing and monitoring to detect this condition in the future.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qJ3IUAU"},"Id":"a1P1Y000005qJ3IUAU","CreatedDate":"2019-07-11T18:48:45.000+0000","IncidentId__c":"a9C1Y000000fxgDUAQ","UpdateLog__c":"Resolved: An issue impacting syslog accessibility in all Okta Cells has been identified and fixed. Service has been restored to normal. Additional root cause information will be provided within 48 business hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qJP9UAM"},"Id":"a1P1Y000005qJP9UAM","CreatedDate":"2019-07-17T06:55:39.000+0000","IncidentId__c":"a9C1Y000000fxgDUAQ","UpdateLog__c":"ROOT CAUSE ANALYSIS: On On July 11th, at approximately 10:47 AM PDT Okta detected System Logs access failures across all cells. \r\n\r\nUpon investigation, Okta determined that the issue was caused by an expired Certificate Revocation List (CRL), which had been updated in the certificate keystore repository, but had not been synced across System Logs storage infrastructure. This resulted in certificate validation failures when users attempted to access System Logs. \r\n\r\nAt 11:06 AM PDT, Okta updated the CRL across all cells. Errors started clearing as the change was applied and caches were progressively refreshed. At approximately 11:28 AM PDT, all errors had been cleared. \r\n\r\nOkta is implementing additional monitoring and alerting to further enhance detection of similar CRL errors in the future. We are also implementing additional process changes and tooling to prevent similar CRL-related certificate validation errors in the future.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qJm8UAE"},"Id":"a1P1Y000005qJm8UAE","CreatedDate":"2019-08-19T20:32:00.000+0000","IncidentId__c":"a9C1Y000000fxh6UAA","UpdateLog__c":"Resolved: On Aug 19th at approximately 12:30PM PDT, Okta noted an issue affecting GSuite's OAuth 2.0 endpoints preventing customers who use GSuite for authentication from accessing ASA's web interface or enrolled servers, Google imports, and SSO login to GSuite apps.\r\n\r\nGoogle is reporting the issue is resolved for OAuth 2.0.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qJosUAE"},"Id":"a1P1Y000005qJosUAE","CreatedDate":"2019-08-22T05:27:49.000+0000","IncidentId__c":"a9C1Y000000fxh6UAA","UpdateLog__c":"Root Cause Analysis: On August 19th, at approximately 10:50 AM PDT Okta detected an increased number of failed Google OAuth 2.0 authentication requests and import failures. In addition to that, customers using GSuite as their main IdP could not authenticate via Advanced Server Access (ASA) to access their infrastructure. \r\n\r\nUpon investigation, Okta determined that Google had experienced a global outage of their OAuth 2.0 endpoints. This prevented external services (ASA, Okta) from authenticating user sessions against Google as an IdP, and no workarounds were provided. \r\n\r\nAt approximately 12:30 PM PDT the issue with Google OAuth 2.0 endpoints was fully resolved and service was restored.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qK3nUAE"},"Id":"a1P1Y000005qK3nUAE","CreatedDate":"2019-09-05T19:56:49.000+0000","IncidentId__c":"a9C1Y000000fxj2UAA","UpdateLog__c":"Okta has identified the cause of the Activation Email regression affecting new Developer tenants and is currently preparing to deploy a fix.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qK3sUAE"},"Id":"a1P1Y000005qK3sUAE","CreatedDate":"2019-09-05T20:36:07.000+0000","IncidentId__c":"a9C1Y000000fxj2UAA","UpdateLog__c":"Okta has developed and verified the fix, and is working to deploy the fix to US Cell 7.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qK3xUAE"},"Id":"a1P1Y000005qK3xUAE","CreatedDate":"2019-09-05T21:15:18.000+0000","IncidentId__c":"a9C1Y000000fxj2UAA","UpdateLog__c":"Okta has successfully deployed and verified the fix for the issue with new Developer tenant activation e-mail in US Cell 7. Additional root cause information will provided within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000005qK7eUAE"},"Id":"a1P1Y000005qK7eUAE","CreatedDate":"2019-09-10T06:20:40.000+0000","IncidentId__c":"a9C1Y000000fxj2UAA","UpdateLog__c":"ROOT CAUSE: On September 5th, at approximately 12:27 PM PDT Okta detected an issue in the Developer tenant's activation email which impacted developer org creation in US Preview 1 and US Cell 7. \r\n\r\nUpon investigation, Okta determined that a software change to the Developer Tenant Email Template caused the generated emails to have noticeable styling issues and, in some clients, this caused one-time temporary passwords to become unreadable. \r\n\r\nAt approximately 1:49 PM PDT Okta verified a fix and began deploying the change as a patch to the impacted cells. At approximately 2:13 PM PDT the fix was fully deployed and generated emails were sent with visible temporary passwords. \r\n\r\nAdditional testing and monitoring to detect and mitigate this scenario will be rolled out for affected environments to prevent future occurrences.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHjoUAE"},"Id":"a1P1Y000007zHjoUAE","CreatedDate":"2019-09-30T23:20:43.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"We’ve narrowed the issue down to Google Chrome version 77 (on both desktop and Android). This is due to a change introduced in Google Chrome version 77. We are working on a workaround to address the impact of slow page load for the Okta Dashboard.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHjtUAE"},"Id":"a1P1Y000007zHjtUAE","CreatedDate":"2019-10-01T00:07:15.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"We are continuing to work on a workaround. \r\n\r\nPlease note that this issue is specific to Google Chrome version 77 and does not impact Mozilla Firefox, Internet Explorer, or Apple Safari.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHjyUAE"},"Id":"a1P1Y000007zHjyUAE","CreatedDate":"2019-10-01T00:43:45.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Our team is continuing to work on the workaround, and we should be able to start deploying it soon.\r\n\r\nNote that this issue impacts Chromium-based browsers, so this issue may also occur if you’re using the Chromium-based preview version of Microsoft Edge. Non-Chromium-Based versions of Microsoft Edge are not impacted.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHk3UAE"},"Id":"a1P1Y000007zHk3UAE","CreatedDate":"2019-10-01T01:11:23.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Our team is continuing to work on the workaround to resolve this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHkDUAU"},"Id":"a1P1Y000007zHkDUAU","CreatedDate":"2019-10-01T01:48:25.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Our team is continuing to work on the workaround to resolve this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHkIUAU"},"Id":"a1P1Y000007zHkIUAU","CreatedDate":"2019-10-01T01:57:22.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Okta will begin deployment tonight and expect to complete the deployment to all cells by Oct 1 2019 7:00 pm PDT.\r\n\r\nUsers who cannot rely on a browser other than Chrome to use Okta, may workaround the issue by disabling Hardware Acceleration in Chrome Advanced/System Preferences and relaunching browser\r\n\r\nOkta is working with Google to confirm the suspected browser issue resolution\r\n\r\nWe will continue to update as the workaround is deployed to specific Cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHkSUAU"},"Id":"a1P1Y000007zHkSUAU","CreatedDate":"2019-10-01T03:49:19.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Workaround has been deployed to Cell EU1. We will continue to update as the workaround is deployed to other cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHkXUAU"},"Id":"a1P1Y000007zHkXUAU","CreatedDate":"2019-10-01T05:06:40.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Workaround has been deployed to Cell OK1. We will continue to update as the workaround is deployed to other cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHnRUAU"},"Id":"a1P1Y000007zHnRUAU","CreatedDate":"2019-10-01T19:38:20.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Workaround has been deployed to Cell OK8 and Ok9. We will continue to update as the workaround is deployed to the remaining cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHnbUAE"},"Id":"a1P1Y000007zHnbUAE","CreatedDate":"2019-10-01T21:26:19.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Workaround has been deployed to Cell OK5. We will continue to update as the workaround is deployed to the remaining cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHngUAE"},"Id":"a1P1Y000007zHngUAE","CreatedDate":"2019-10-01T22:27:04.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Workaround has been deployed to Cell OK2. We will continue to update as the workaround is deployed to the remaining cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHnlUAE"},"Id":"a1P1Y000007zHnlUAE","CreatedDate":"2019-10-01T23:06:47.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Workaround has been deployed to Cell OK3. We will continue to update as the workaround is deployed to other cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHnqUAE"},"Id":"a1P1Y000007zHnqUAE","CreatedDate":"2019-10-01T23:39:50.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Workaround has been deployed to Cell OK4. We will continue to update as the workaround is deployed to other cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHnvUAE"},"Id":"a1P1Y000007zHnvUAE","CreatedDate":"2019-10-02T00:29:07.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Workaround has been deployed to Cell OK7. We will continue to update as the workaround is deployed to other cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHo0UAE"},"Id":"a1P1Y000007zHo0UAE","CreatedDate":"2019-10-02T01:40:47.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Resolved: The issue with Google Chrome version 77 (on both desktop and Android) which was due to a change introduced in Google Chrome version 77 is now resolved on all Cells. Additional root cause information will be provided within 48 business hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zHpmUAE"},"Id":"a1P1Y000007zHpmUAE","CreatedDate":"2019-10-04T02:10:12.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Root Cause Analysis: On Sept 30th, at approximately 3:39PM PDT, customers reported experiencing longer than expected load times for application logos in the Okta end-user dashboard and catalog.\r\n\r\nUpon investigation, Okta determined users in all cells experienced the longer load times after updating their Chrome browser to version 77.0.3865.75 (Official Build) (64-bit). Users of browsers other than Chrome, and users of Chrome earlier than version 77, were not impacted. Okta discovered that Chrome version 77 caused a performance regression with some HTML Canvas operations when the hardware acceleration setting was available and enabled. This behavior is also discussed in the Chromium bug forum as Issue 1001845.\r\n\r\nAt approximately 8:40PM PDT on Sept 30th, Okta started the phased deployment of a workaround, per-cell basis, which was completed on all cells by 10/1/2019 at 6:40 PM PDT.\r\n\r\nDuring the incident, users were advised to workaround the issue by disabling Hardware Acceleration. This workaround is no longer necessary and Hardware Acceleration in Chrome can be re-enabled.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zIozUAE"},"Id":"a1P1Y000007zIozUAE","CreatedDate":"2019-11-06T18:23:00.000+0000","IncidentId__c":"a9C1Y0000004E2wUAE","UpdateLog__c":"Okta is currently experiencing an issue with Office 365 Sign on for a subset of users. We are actively investigating and a possible fix is being proposed to address this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zIpOUAU"},"Id":"a1P1Y000007zIpOUAU","CreatedDate":"2019-11-06T18:49:41.000+0000","IncidentId__c":"a9C1Y0000004E2wUAE","UpdateLog__c":"Okta has successfully tested the fix in OK4 Cell. We are working diligently to apply the fix to all the Okta Cells.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zIpTUAU"},"Id":"a1P1Y000007zIpTUAU","CreatedDate":"2019-11-06T19:29:05.000+0000","IncidentId__c":"a9C1Y0000004E2wUAE","UpdateLog__c":"RESOLVED: Okta has successfully deployed and verified a fix across all Cells. The identified issue impacted a subset of the users in all cells using legacy protocol sign-on with Office 365 or service accounts. Additional root cause information will provided within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zIsSUAU"},"Id":"a1P1Y000007zIsSUAU","CreatedDate":"2019-11-10T17:26:13.000+0000","IncidentId__c":"a9C1Y0000004E2wUAE","UpdateLog__c":"Root Cause Analysis:\r\n\r\nOn Nov 6th at approximately 9:30 AM PDT, Okta received reports from certain customers across multiple cells that some users were not able to log into the Okta Service via Office 365 clients using legacy protocols such as the Outlook Desktop App, and some service accounts used for automation.\r\n\r\nOnly end users using Outlook clients with an Exchange ActiveSync profile and without an enrolled second factor that are members of tenants with a configured MFA Enrollment Policy were impacted.\r\n\r\nThe pursuant investigation determined that a bug fix, applied in the 2019.10.2 weekly update, to address a gap in multi-factor enrollment during authentication, was the root cause. The fix was incorrectly scoped to include legacy Office 365 protocols.\r\n\r\nAt approximately 10:44 AM PDT, Okta reverted the change in US Cell 4 and confirmed that issue was resolved. The change was then subsequently reverted in all production. At approximately 11:35 AM, service was fully restored for all customers in all cells.\r\n\r\nAdditional testing and alerting to prevent and detect such issues will be implemented to prevent future occurrences.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJMrUAM"},"Id":"a1P1Y000007zJMrUAM","CreatedDate":"2019-11-19T18:32:32.000+0000","IncidentId__c":"a9C1Y0000004E4EUAU","UpdateLog__c":"Okta customers in all cells that leverage Zoom’s integration for provisioning are experiencing errors while updating users in Zoom. We are still working with Zoom to investigate the errors and will provide another update in 2 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJOYUA2"},"Id":"a1P1Y000007zJOYUA2","CreatedDate":"2019-11-19T20:35:05.000+0000","IncidentId__c":"a9C1Y0000004E4EUAU","UpdateLog__c":"Zoom confirmed that a recent release to add an email update feature resulted in a change to the provisioning API. This change has impacted Okta customers in all cells that leverage Okta's Zoom integration for provisioning - customers are seeing errors while updating users in Zoom. We are continuing to work with Zoom to identify a solution and timeline. We will provide the next update in 2 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJOdUAM"},"Id":"a1P1Y000007zJOdUAM","CreatedDate":"2019-11-19T22:00:06.000+0000","IncidentId__c":"a9C1Y0000004E4JUAU","UpdateLog__c":"Okta is able to reproduce, and continues to investigate, an intermittent issue presenting users with the error “Detected possible malicious activity when establishing a secure connection to Okta” when installing or opening the Okta Mobile application on Android. We will provide more information once available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJOiUAM"},"Id":"a1P1Y000007zJOiUAM","CreatedDate":"2019-11-19T22:34:57.000+0000","IncidentId__c":"a9C1Y0000004E4JUAU","UpdateLog__c":"Okta was able to reproduce an intermittent issue presenting users with the error “Detected possible malicious activity when establishing a secure connection to Okta” when installing or opening the Okta Mobile application on Android. We are continuing to investigate and will provide more information once available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJOnUAM"},"Id":"a1P1Y000007zJOnUAM","CreatedDate":"2019-11-19T23:04:32.000+0000","IncidentId__c":"a9C1Y0000004E4EUAU","UpdateLog__c":"Okta continues to work with Zoom to resolve this issue. We are actively working to deploy a fix to allow provisioning to complete without error. Zoom has in-tandem confirmed they will be working implement a fix for the underlying issue later today.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJOsUAM"},"Id":"a1P1Y000007zJOsUAM","CreatedDate":"2019-11-19T23:10:01.000+0000","IncidentId__c":"a9C1Y0000004E4JUAU","UpdateLog__c":"Investigation is underway for the intermittent issue that Okta reproduced. Customers may see the following error: “Detected possible malicious activity when establishing a secure connection to Okta” when installing or opening the Okta Mobile application on Android. We will provide more information once available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJOxUAM"},"Id":"a1P1Y000007zJOxUAM","CreatedDate":"2019-11-19T23:55:49.000+0000","IncidentId__c":"a9C1Y0000004E4JUAU","UpdateLog__c":"Okta has determined the root cause of the issue on Android when installing or launching Okta Mobile or Okta Verify. Okta Engineering is actively working on a fix and will release new versions of Okta Mobile and Okta Verify as soon as they are ready. While the message displayed in the app may sound alarming, the issue is benign, and due to our over abundance of caution with TLS algorithm restrictions.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJP2UAM"},"Id":"a1P1Y000007zJP2UAM","CreatedDate":"2019-11-20T00:41:22.000+0000","IncidentId__c":"a9C1Y0000004E4JUAU","UpdateLog__c":"Okta has determined the root cause of the issue on Android when installing or launching Okta Mobile or Okta Verify. Okta Engineering is actively working on a fix and will release new versions of Okta Mobile and Okta Verify as soon as they are ready. While the message displayed in the app may sound alarming, the issue is benign, and due to our over abundance of caution with TLS algorithm restrictions. The next update will be posted in approximately 2 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJP7UAM"},"Id":"a1P1Y000007zJP7UAM","CreatedDate":"2019-11-20T03:05:37.000+0000","IncidentId__c":"a9C1Y0000004E4JUAU","UpdateLog__c":"Okta has determined the root cause of the issue on Android when installing or launching Okta Mobile or Okta Verify. Okta Engineering is actively working on a fix and expects to begin roll-out of the new versions of Okta Mobile and Okta Verify within the next hour. While the message displayed in the app may sound alarming, the issue is benign, and due to our over abundance of caution with TLS algorithm restrictions.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJPCUA2"},"Id":"a1P1Y000007zJPCUA2","CreatedDate":"2019-11-20T04:30:22.000+0000","IncidentId__c":"a9C1Y0000004E4JUAU","UpdateLog__c":"Okta has determined the root cause of the issue on Android when installing or launching Okta Mobile. Okta Engineering is actively working on a fix and expects to release the new version of Okta Mobile within the hour. Okta has determined there is no meaningful impact to Okta Verify at this point. Additionally, while the message displayed in the Okta Mobile app may sound alarming, the issue is benign, and due to our over abundance of caution with TLS algorithm restrictions.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJPHUA2"},"Id":"a1P1Y000007zJPHUA2","CreatedDate":"2019-11-20T05:02:16.000+0000","IncidentId__c":"a9C1Y0000004E4EUAU","UpdateLog__c":"Resolved: An issue impacting user provisioning updates to the Zoom application within Okta has been resolved. During the event, user updates to the Zoom application may have encountered errors if the provisioned user already existed within Zoom. Okta worked with Zoom to identify the root cause of the issue and applied a hot-fix to all production cells to mitigate the provisioning errors while Zoom worked to deploy a permanent fix. Additional root cause information will be provided within 48 business hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJPMUA2"},"Id":"a1P1Y000007zJPMUA2","CreatedDate":"2019-11-20T06:00:37.000+0000","IncidentId__c":"a9C1Y0000004E4JUAU","UpdateLog__c":"Resolved: Okta has addressed an issue on Okta Mobile for Android for the vast majority of Okta end users. When opening the Okta Mobile for Android application, end users may be logged out of their account and encounter the following message: “Detected possible malicious activity when establishing a secure connection to Okta”. Okta Mobile Android users on Android 6.0 or higher must update the Okta Mobile application to the latest version through the Google Play Store. Okta continues to investigate a solution for users running versions of Android below 6.0 as well as those relying on downloading the Okta Mobile APK from their Okta tenant. We apologize for the inconvenience. Additional root cause information will be provided within 48 business hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJUvUAM"},"Id":"a1P1Y000007zJUvUAM","CreatedDate":"2019-11-22T16:15:09.000+0000","IncidentId__c":"a9C1Y0000004E4JUAU","UpdateLog__c":"Root Cause Analysis: On Nov 19th at approximately 12:30 PM PDT, customers reported that when opening the Okta Mobile application for Android, end users were intermittently logged out of their account and encountered the following message:\r\n\r\n“Detected possible malicious activity when establishing a secure connection to Okta”.\r\n\r\nUpon investigation, Okta determined that the issue was caused by an update to the preferred authentication algorithm used in TLS by Okta Mobile’s error tracking provider. The new preferred authentication algorithm chosen by the provider became ECDSA, which was not allowed by the Okta Mobile Android application. Upon detecting a change in the encryption algorithm in use, Okta Mobile on Android is designed to present the aforementioned warning message.\r\n\r\nOn Nov 19th at approximately 8:45 PM PDT, Okta released Android Okta Mobile version 4.3.2, for Android versions 6.0 and above, and notified Org administrators through the Admin Console banner that users should update their Okta Mobile application to address the issue. Users running an older version of Android are encouraged to update their device to 6.0 or later and install the latest version of Okta Mobile.\r\n\r\nOkta is implementing additional monitoring and alerting to detect such issues earlier in the future. Additionally, the latest Android Okta Mobile version has expanded the set of authentication algorithms allowed and newer versions will include an improved design to prevent issues like this in the future.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJV0UAM"},"Id":"a1P1Y000007zJV0UAM","CreatedDate":"2019-11-22T17:31:58.000+0000","IncidentId__c":"a9C1Y0000004E4EUAU","UpdateLog__c":"Root Cause Analysis: On Nov 19th at approximately 8:53AM PDT, customers reported Zoom provisioning errors when updating user’s profiles.\r\n\r\nUpon investigation, Okta determined that the issue was caused by a change in the behavior of an existing Zoom Provisioning API which was introduced on November 17. This change was implemented by Zoom to enable support for changing user's email addresses through the Profile Update request. The change inadvertently caused profile updates to fail, whenever the email has not been changed.\r\n\r\nOn Nov 19th at approximately 8:17 PM PDT Okta released a work around to address the issue. This update was deployed to all production cells and service was fully restored.\r\n\r\nOkta is working closely with Zoom for a fix, as well as, understanding and mitigating the re-occurrence for this type of failure","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJVeUAM"},"Id":"a1P1Y000007zJVeUAM","CreatedDate":"2019-11-23T23:51:04.000+0000","IncidentId__c":"a9C1Y0000004E5RUAU","UpdateLog__c":"Okta is continuing to investigate the root cause of the the issue and is taking steps to mitigate the issue. Some Administrators and Users are still experiencing performance degradation, and we are continuing to work to resolve this as quickly as possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJVjUAM"},"Id":"a1P1Y000007zJVjUAM","CreatedDate":"2019-11-24T00:52:43.000+0000","IncidentId__c":"a9C1Y0000004E5RUAU","UpdateLog__c":"Okta is continuing to investigate the root cause of the the issue and is taking steps to mitigate the issue. The issue is still ongoing and we are continuing to work to resolve this as quickly as possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJVoUAM"},"Id":"a1P1Y000007zJVoUAM","CreatedDate":"2019-11-24T01:25:53.000+0000","IncidentId__c":"a9C1Y0000004E5RUAU","UpdateLog__c":"Okta has taken steps to mitigate the issue, and we are seeing improvements in Dashboard and Authentication requests. At this time user provisioning has been disabled, we are working to restore this functionality and investigating the root cause. Further updates will be provided as we continue to progress towards full resolution","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJVtUAM"},"Id":"a1P1Y000007zJVtUAM","CreatedDate":"2019-11-24T02:01:58.000+0000","IncidentId__c":"a9C1Y0000004E5RUAU","UpdateLog__c":"Okta has taken steps to mitigate the issue, and we are seeing improvements in Dashboard and Authentication requests. At this time user provisioning is still disabled as we work through the remaining issues. Further updates will be provided as we continue to progress towards full resolution","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJVyUAM"},"Id":"a1P1Y000007zJVyUAM","CreatedDate":"2019-11-24T02:57:49.000+0000","IncidentId__c":"a9C1Y0000004E5RUAU","UpdateLog__c":"Resolved: An issue impacting Okta Advanced Server Access has been resolved. During the event, Administrators may have experienced errors accessing the ASA dashboard. Additionally, users may have experienced authentication errors. Okta applied several changes to mitigate the issue and continues to monitor and investigate. Additional root cause information will be provided within 48 business hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJhQUAU"},"Id":"a1P1Y000007zJhQUAU","CreatedDate":"2019-12-02T17:12:06.000+0000","IncidentId__c":"a9C1Y0000004E5WUAU","UpdateLog__c":"Okta is seeing delays in processing of jobs within a subset of our job framework in US Cell 7. We are actively working to mitigate job processing delays and will continue to provide updates as they are available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJhVUAU"},"Id":"a1P1Y000007zJhVUAU","CreatedDate":"2019-12-02T17:45:09.000+0000","IncidentId__c":"a9C1Y0000004E5WUAU","UpdateLog__c":"Okta is continuing to work to mitigate the job processing delays for a subset of customers in US Cell 7. Queued jobs are processing, but at a reduced rate. We will provide our next update by 10:15am PST.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJhaUAE"},"Id":"a1P1Y000007zJhaUAE","CreatedDate":"2019-12-02T18:23:57.000+0000","IncidentId__c":"a9C1Y0000004E5WUAU","UpdateLog__c":"An issue impacting job processing for a subset of customers within US Cell 7 has been resolved. The remaining queued jobs are processing at full capacity and we will continue to monitor. Root Cause Analysis is underway and we will post full RCA details here within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJiEUAU"},"Id":"a1P1Y000007zJiEUAU","CreatedDate":"2019-12-03T21:50:03.000+0000","IncidentId__c":"a9C1Y0000004E5RUAU","UpdateLog__c":"Root Cause Analysis: On Saturday, November 23rd at approximately 1:05 PM PDT, Okta detected database resource contention within our Advanced Server Access production environment, resulting in timeout errors for users who were accessing the Advanced Server Access dashboard or were attempting to authenticate to servers.\r\n\r\nAfter investigating the issue, at approximately 3:30 PM PDT, Okta took actions to mitigate the resource contention issue by blocking customer API requests for adding groups to projects and synchronizing data to servers. Most Advanced Server Access functionality was restored through this action. Upon further investigation, Okta identified an additional inefficiency in our synchronous logic, and made a change to asynchronous background job processing.\r\n\r\nFollowing our investigation and mitigation actions, Okta re-enabled the API for synchronizing data to servers at approximately 6:47 PM PDT. Okta also re-enabled the API for adding groups to projects, fully restoring all services at this time.\r\n\r\nOkta's has implemented changes to mitigate the impact of similar issues in the future, and is continuing its full Root Cause Analysis of the incident,. We expect to be able to provide further details and future preventative actions by December 4, 2019. Additional RCA details will be posted here once this is investigation is complete.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJinUAE"},"Id":"a1P1Y000007zJinUAE","CreatedDate":"2019-12-04T21:21:42.000+0000","IncidentId__c":"a9C1Y0000004E65UAE","UpdateLog__c":"Okta has identified the cause of the Workday import triggering users to be placed in No Password state in US Cell 3. We are actively working to mitigate the issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJisUAE"},"Id":"a1P1Y000007zJisUAE","CreatedDate":"2019-12-04T21:37:14.000+0000","IncidentId__c":"a9C1Y0000004E65UAE","UpdateLog__c":"Okta has resolved the issue where Workday imports for some Orgs changed users to a No Password state resulting in those users being unable to authenticate. Affected customers must rerun a Workday import to return affected end-users to Active state.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJixUAE"},"Id":"a1P1Y000007zJixUAE","CreatedDate":"2019-12-04T22:22:54.000+0000","IncidentId__c":"a9C1Y0000004E65UAE","UpdateLog__c":"Okta has resolved the issue where Workday imports for some Orgs changed users to a No Password state resulting in those users being unable to authenticate. Root Cause Analysis will be provided here within 2 business days.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJkPUAU"},"Id":"a1P1Y000007zJkPUAU","CreatedDate":"2019-12-08T03:04:04.000+0000","IncidentId__c":"a9C1Y0000004E5WUAU","UpdateLog__c":"Root Cause Analysis: On December 2nd at approximately 7:45AM (PST), Okta detected a slowdown with processing import jobs on US Cell 7 and subsequently determined that there was an unusual and significant increase in asynchronous job workload which resulted in a backup of job processing.\r\n\r\nOkta increased system resources to expedite the processing of pending jobs. This mitigated the issue and normal functionality was fully restored by ‪10:16 AM.\r\n\r\nOkta is working optimize the current implementation of group application assignments to batch and de-duplicate assignments. Okta is also adding monitoring and controls to detect and mitigate similar issues in the future","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJkUUAU"},"Id":"a1P1Y000007zJkUUAU","CreatedDate":"2019-12-08T04:10:43.000+0000","IncidentId__c":"a9C1Y0000004E65UAE","UpdateLog__c":"Root Cause Analysis: On December 4th 2019 at 11:16am PST, customers on US Cell 3 began reporting issues with authentication and user provisioning after Workday imports. Upon investigation, Okta determined that a subset of customers on US Cell 3 who imported groups to Okta from Workday, Ultipro, or ServiceNow after 10am PST experienced a loss of group membership status for some users after import.\r\n\r\nIn cases where group membership was tied to provisioning for downstream applications, the loss of group memberships also removed user access to those applications. For impacted customers that provisioned Workday users to Active Directory (AD), where group members were part of delegated authentication to AD, affected users were unable to log into Okta.\r\n\r\nAt approximately 1:10pm PST, Okta took action to prevent further customer impact by disabling user import jobs on US Cell 3. Additional investigation determined the observed behavior was a result of an in-progress infrastructure change for existing customers in US Cell 3, which began rolling out at approximately 10am PST. At 1:20pm PST Okta took action to roll back this change and at 1:30 PST import jobs were re-enabled on US Cell 3. Following these updates, Okta published remediation steps and worked directly with the small subset of customers experiencing on-going impact to fully restore user provisioning and group assignments.\r\n\r\nSite functionality was restored at 2:20pm PST. However, additional action was required for a small subset of customers to restore functionality for downstream applications.\r\n\r\nInfrastructure changes were deployed as a phased roll-out within Preview Cells starting March 2019 and enabled for new customers across Production Cells on November 4, 2019. After several months of validation with no issues reported, these changes were approved for deployment on US Cells with existing customers. Despite pre-release testing and a phased roll-out approach to mitigate risk for existing customers, Okta had a testing gap for migrating existing customers running import jobs for group assignment.\r\n\r\nOkta is rigorously reviewing test coverage for import scenarios and working directly with customers to further identify complex use cases with downstream dependencies to enhance our test suites. As part of our on-going focus on reliability, Okta is also working to implement more granular monitoring and automated mitigation mechanisms to protect against similar issues during future infrastructure updates.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJmVUAU"},"Id":"a1P1Y000007zJmVUAU","CreatedDate":"2019-12-10T18:40:31.000+0000","IncidentId__c":"a9C1Y0000004E6jUAE","UpdateLog__c":"Okta is currently experiencing an issue with Agentless Desktop SSO for a subset of customers in US Cell 1, US Cell 11, and EMEA Cell 1 resulting in users being redirected to the default login page rather then being authenticated via Agentless Desktop SSO. We are currently taking mitigating actions for the impacted customers","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJmaUAE"},"Id":"a1P1Y000007zJmaUAE","CreatedDate":"2019-12-10T19:09:02.000+0000","IncidentId__c":"a9C1Y0000004E6jUAE","UpdateLog__c":"Resolved: Okta has resolved an issue where a subset of customers were unable to authenticate successfully via Agentless Desktop SSO in US Cell 1, US Cell 11, and EMEA Cell 1. Impacted customers may be required to clear their DNS cache to fully restore end user normalcy. Root Cause Analysis will be provided here within 2 business days.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJqIUAU"},"Id":"a1P1Y000007zJqIUAU","CreatedDate":"2019-12-16T08:19:41.000+0000","IncidentId__c":"a9C1Y0000004E6jUAE","UpdateLog__c":"Root Cause Analysis: On December 10th, 2019 at approximately 9:30 AM PST, Okta began investigating reports by customers of problems authenticating via Agent-less Desktop SSO in US Cell 1, US Cell 11, and EU Cell 1. \r\n\r\nOkta determined that during the implementation of new load-balancers on December 4th 2019 at 3:15pm PST, DNS entries for Kerberos endpoints were not updated correctly for the impacted cells. As a result, Agent-less Desktop SSO traffic was still directed to the previous load balancers which were no longer working as intended, resulting in requests that timed out and user browser sessions being redirected to Okta login screens. While affected users were able to login, they had to type in their credentials instead of being logged in automatically.\r\n\r\nAt approximately 10:48 AM, Okta updated the affected entries to reference the new load-balancers and service was restored. Some clients may have had to refresh DNS caches to restore service more quickly.\r\n\r\nOkta is working to implement additional monitoring and automation of changes to DNS records to prevent similar issues during future updates.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zJtgUAE"},"Id":"a1P1Y000007zJtgUAE","CreatedDate":"2019-12-19T02:29:57.000+0000","IncidentId__c":"a9C1Y0000004E73UAE","UpdateLog__c":"Root Cause Analysis: On December 16, 2019 at approximately 6:22 AM PST, Okta detected a degradation in application performance in US Cell 4. As a result, customers may have experienced:\r\n- Longer than normal response times\r\n- A small percentage (0.25%) experienced sporadic read-only errors\r\n- Admin users may have intermittently seen the read-only banner\r\n\r\nUpon investigation, Okta determined that the root cause was unusual database activity required to process a significant number of group rules and assignments. Although our throttling and rate limiting mechanisms engaged, a portion of requests saw additional latency due to the increased utilization of our database tier while the load was being redistributed.\r\nAt approximately 6:48 AM PST, Okta began to redistribute requests across the cell’s database cluster to reduce load and contention on the impacted database. By 8:09 AM PST, normal service performance was fully restored.\r\n\r\nOkta is implementing additional monitoring, optimization, and mitigation strategies to prevent similar issues in the future. Okta is also investigating asynchronous processing for group rules and performance improvements for group deletions to prevent system slowdown.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zK0rUAE"},"Id":"a1P1Y000007zK0rUAE","CreatedDate":"2020-01-08T14:18:37.000+0000","IncidentId__c":"a9C1Y0000004E7hUAE","UpdateLog__c":"Root Cause Analysis: On January 5th at approximately 1:19am PST Okta received an alert that an automated fail-over had occurred within our caching infrastructure in US Cell 4.\r\n\r\nInitial investigation determined that a cache infrastructure server in one availability zone had experienced a brief network outage. As a result, a secondary cache node in a different availability zone was automatically promoted to primary. Manual analysis of slot and node health indicated that all nodes in the cluster were healthy but that cluster resiliency was degraded as a result of the fail-over.\r\n\r\nIn order to regain cluster resiliency, a fail-back command to revert to the original primary/secondary node relationship was manually issued at 1:39am PST. This fail-back attempt completed. However, immediately thereafter all requests to the specific hash range served by this node either timed out if the primary was contacted directly or errored out if the request hit another node of the cluster.\r\n\r\nThe combination of timeouts, retries, and new requests resulted in customers in US Cell 4 experiencing increased latency and intermittent 429 (rate limit) or 502 (server resource unavailable) error responses.\r\n\r\nFurther investigation determined the original network outage and corresponding secondary node promotion was due to a local hardware issue on the cache cluster primary node from the underlying cloud network service provider. Cluster health checks performed prior to fail-back did not capture this underlying hardware instability, which was on-going. Due to this oversight, the fail-back command resulted in a primary node assignment within an unstable environment. After the fail-back to the original but unstable server was complete, the original primary node became unresponsive and was unable to fail-over, either automatically or manually.\r\n\r\nSuch a scenario was not covered by our Operational Run-books, and a cache cluster specialist was engaged. At approximately 2:17 AM PST, Okta succeeded in initiating a forced fail-over within US Cell 4 to regain cluster stability and restore service. Service was fully restored at 2:38 AM PST.\r\n\r\nAs a result of this outage, Okta is upgrading run-books to cover non-responsive primaries and is evaluating training, specialization, and follow the sun coverage to resolve specific cache cluster recovery procedures when automatic fail-over is not possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zK4PUAU"},"Id":"a1P1Y000007zK4PUAU","CreatedDate":"2020-01-23T00:06:16.000+0000","IncidentId__c":"a9C1Y0000004E8uUAE","UpdateLog__c":"Okta is investigating to restore the incorrect AD Agent state information being reported. Customers with IWA automatic fail-over failing can change to manual fail-over mode to return IWA authentication to service","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zK4UUAU"},"Id":"a1P1Y000007zK4UUAU","CreatedDate":"2020-01-23T00:43:57.000+0000","IncidentId__c":"a9C1Y0000004E8uUAE","UpdateLog__c":"Okta is actively working to restore the incorrect AD Agent state information being reported. While being reported as NOT_CONNECTED, monitoring shows that AD Agent authentication continues to be successful.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zK4ZUAU"},"Id":"a1P1Y000007zK4ZUAU","CreatedDate":"2020-01-23T01:16:26.000+0000","IncidentId__c":"a9C1Y0000004E8uUAE","UpdateLog__c":"Okta has resolved an issue in US Cell 1 impacting AD/LDAP Agent state reporting. All on-line AD/LDAP Agents are now correctly reporting ACTIVE status. Additionally, some customers where all AD/LDAP Agents were marked NOT_CONNECTED should now see successful imports. Additional root cause information will be provided within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zK5wUAE"},"Id":"a1P1Y000007zK5wUAE","CreatedDate":"2020-01-27T20:17:34.000+0000","IncidentId__c":"a9C1Y0000004E8uUAE","UpdateLog__c":"Root Cause Analysis: On January 22nd at approximately ‪11:39am PST‬ Okta’s monitoring systems detected internal server errors in US Cell 1. The errors resulted in some customers' AD and LDAP agents being reported as NOT_CONNECTED to Okta Admins, even though the agents were functioning normally. Additionally, customers where all AD Agents were marked offline and who had Desktop Single Sign-On (IWA) configured using automatic fail-over may have experienced IWA failures. Okta implemented mitigation steps and the errors subsided at approximately ‪12:06pm PST‬. Okta subsequently corrected agent status for all customers by approximately ‪5:00pm PST‬.\r\n\r\nUpon investigation, Okta determined that a mistake was made when updating references to the database system during the execution of a planned routine database maintenance, This misconfiguration caused a very small number of requests to return an HTTP 500 error code and also resulted in the outdated agent status information.\r\n\r\nOkta has updated run-books and is improving the automation of maintenance verification procedures to prevent future occurrences. Okta is also investigating detection of agent status check job failures to prevent incorrect status indication to administrators.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKafUAE"},"Id":"a1P1Y000007zKafUAE","CreatedDate":"2020-02-25T22:25:18.000+0000","IncidentId__c":"a9C1Y0000004EBeUAM","UpdateLog__c":"Okta is continuing to work on resolving an issue with delayed processing of user import jobs in Okta Preview Cell 1, US Cell 6 and US Cell 7. We are implementing mitigating steps to return the service to normal as quickly as possible.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKakUAE"},"Id":"a1P1Y000007zKakUAE","CreatedDate":"2020-02-25T23:00:07.000+0000","IncidentId__c":"a9C1Y0000004EBeUAM","UpdateLog__c":"Okta has addressed the delayed processing of user import jobs in US Cell 6 and US Cell 7. We will continue to monitor the progress to ensure services have returned to normal operations.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKapUAE"},"Id":"a1P1Y000007zKapUAE","CreatedDate":"2020-02-25T23:11:00.000+0000","IncidentId__c":"a9C1Y0000004EBeUAM","UpdateLog__c":"Resolved: Okta has addressed the delayed user import jobs issue on Okta Preview 1, US Cell 6, and US Cell 7. We will continue to monitor and additional root cause information will be provided within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKbTUAU"},"Id":"a1P1Y000007zKbTUAU","CreatedDate":"2020-02-27T20:10:49.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta has reproduced the issue and is working towards a resolution.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKbsUAE"},"Id":"a1P1Y000007zKbsUAE","CreatedDate":"2020-02-27T20:39:02.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta is continuing work towards implementing a fix to address the issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKc2UAE"},"Id":"a1P1Y000007zKc2UAE","CreatedDate":"2020-02-27T21:45:46.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta is working to restore this functionality. Until functionality is restored customers may not be able to modify or validate configurations on the Amazon AWS integration.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKc7UAE"},"Id":"a1P1Y000007zKc7UAE","CreatedDate":"2020-02-27T22:23:59.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta is continuing to work towards a fix that will resolve this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKcCUAU"},"Id":"a1P1Y000007zKcCUAU","CreatedDate":"2020-02-27T22:55:32.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta is continuing to work towards a fix that will resolve this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKcHUAU"},"Id":"a1P1Y000007zKcHUAU","CreatedDate":"2020-02-27T23:34:03.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta is continuing to work towards a fix that will resolve this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000007zKcMUAU"},"Id":"a1P1Y000007zKcMUAU","CreatedDate":"2020-02-28T00:02:21.000+0000","IncidentId__c":"a9C1Y0000004EBtUAM","UpdateLog__c":"Okta is continuing to work towards a fix that will resolve this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRDaUAO"},"Id":"a1P1Y000009QRDaUAO","CreatedDate":"2020-11-26T03:44:15.000+0000","IncidentId__c":"a9C1Y0000008PepUAE","UpdateLog__c":"The processing delays are improving but we expect the impact to persist for some time. We are monitoring to ensure full mitigation and will provide the next update by 9AM PST.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRDfUAO"},"Id":"a1P1Y000009QRDfUAO","CreatedDate":"2020-11-26T11:41:10.000+0000","IncidentId__c":"a9C1Y0000008PepUAE","UpdateLog__c":"The processing delays have been resolved for OK1, OK2, OK3, and OK4, and System Logs are up-to-date. We are still mitigating processing delays in OP1. We will provide update on OP1 processing by 9AM PST but OK1, OK2, OK3, and OK4 are resolved at this time.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRDkUAO"},"Id":"a1P1Y000009QRDkUAO","CreatedDate":"2020-11-26T17:18:22.000+0000","IncidentId__c":"a9C1Y0000008PepUAE","UpdateLog__c":"Resolved: The processing delays have also been resolved for OP1 and System Logs are up-to-date. Additional root cause information will be provided on Monday, Nov 30th.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRDpUAO"},"Id":"a1P1Y000009QRDpUAO","CreatedDate":"2020-11-27T18:34:35.000+0000","IncidentId__c":"a9C1Y0000008PeuUAE","UpdateLog__c":"We are actively working on a solution for the Okta Verify issue that is impacting users running on iOS 12, which is unsupported. Okta has paused auto-updates to Okta Verify 6.0.1; users on iOS 12 should not manually upgrade to this Okta Verify version. Okta Verify for Android, as well as Okta Verify for iOS 13 and iOS 14, are working as expected. We will provide the next update at 1PM PT.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRDuUAO"},"Id":"a1P1Y000009QRDuUAO","CreatedDate":"2020-11-27T21:18:32.000+0000","IncidentId__c":"a9C1Y0000008PeuUAE","UpdateLog__c":"Update: We have created a new version of Okta Verify that will mitigate the issue for impacted devices running on the iOS 12 operating system. Okta Verify does not officially support iOS 12 but we remain committed to mitigating problems for users who are impacted.\r\n\r\nAdditionally, we are working on further mitigations for all iOS 12 users. We are awaiting Apple’s App Store approval to deploy these solutions. Okta Verify on Android, and iOS 13 & 14, remain unimpacted.\r\n\r\nWe will provide the next update tomorrow (11/28/2020) or as soon as we have approval from Apple.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRDzUAO"},"Id":"a1P1Y000009QRDzUAO","CreatedDate":"2020-11-28T19:13:25.000+0000","IncidentId__c":"a9C1Y0000008PeuUAE","UpdateLog__c":"Resolved: For iOS12 users experiencing crashing on Okta Verify 6.0.1, users may now downgrade to a previously functioning version by fully uninstalling the app and installing version 5.9.0. Detailed instructions for users can be found below: \r\n\r\nIf you are on iOS 12 and Okta Verify 6.0.1, and it crashes regularly, there are two options to restore service.\r\n\r\nIf you do not urgently need to use Okta Verify:\r\n 1) Wait to update to the Okta Verify 6.0.4 release.\r\n 2) This release should be ready as soon as Apple approves it (typically 1-3 business days).\r\n\r\n\r\nIf you urgently need to use Okta Verify (may require support of your admin):\r\n 1) Uninstall Okta Verify 6.0.1.\r\n 2) Install the Okta Verify app again from the App Store. This time you should be able to install the 5.9.0 version of Okta Verify.\r\n 3) If you cannot get into the Okta end-user dashboard to re-enroll Okta Verify, reach out to your administrator and ask them to reset your Okta Verify MFA.\r\n 4) Once your admin resets MFA, you will be able to re-enroll from the Okta Verify 5.9.0 you installed from the App Store.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRE4UAO"},"Id":"a1P1Y000009QRE4UAO","CreatedDate":"2020-12-01T06:13:27.000+0000","IncidentId__c":"a9C1Y0000008PepUAE","UpdateLog__c":"At approximately 6AM PT on Wednesday, November 25th, Okta Engineering was alerted to significant delays in event processing times on US Cell 4. Delays and increased error rates were observed within SysLog event processing and Event Hooks (WebHooks). Investigation revealed one of Okta’s cloud service providers was experiencing a regional outage for a data streaming service. The incident began at 5:15 AM PT and impacted Okta US Cell 1, US Cell 2, US Cell 3, US Cell 4, and US Preview Cell 1.\r\n\r\nDuring the incident Okta customers within impacted cells saw gaps in the events timeline displayed within the SysLog Admin Dashboard UI and returned via the System Log API. Other Okta products and features that leverage the impacted streaming service, such as User Search for entity updates and ThreatInsights for event analysis, remained available throughout the impact window and operated with partially stale data until processing delays were resolved. No data was lost during the incident and all events were processed upon recovery.\r\n\r\nAt approximately 3:15PM PT, error rates from the cloud services vendor improved. Okta responded by devoting resources to speed up event processing throughput and recovery time. These optimizations reduced recovery time by up to 5 hours per cell. User Search data lag was resolved by 6:00PM PT across all cells and event processing recovery times per cell were as follows:\r\n\r\nUS Cell 1: began backfill event processing at 5PM PT and all events were fully processed and populated within the system log dashboard by 10:45PM PT.\r\nUS Cell 2 began backfill event processing at 6:55PM PT and observed all events were fully processed and populated within the system log dashboard by 9:50PM PT.\r\nUS Cell 3 began backfill event processing at 5:00PM PT and all events were fully processed and populated within the system log dashboard by 9:40PM PT.\r\nUS Cell 4 began backfill event processing at 7:20PM PT and all events were fully processed and populated within the system log dashboard at 12:20AM PT on November 26th.\r\nUS Preview Cell 1 began backfill event processing at 7:20PM PT and all events were fully processed and populated within the system log dashboard at 9:00AM PT on November 26th. \r\n\r\nFollowing this impact, Okta has identified potential enhancements to improve recovery times in the event of increased error rates related to data streaming service issues. The design and implementation of such improvements will be added to Okta's infrastructure roadmap.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRETUA4"},"Id":"a1P1Y000009QRETUA4","CreatedDate":"2020-12-02T16:32:17.000+0000","IncidentId__c":"a9C1Y0000008PeuUAE","UpdateLog__c":"Root Cause Analysis: \r\nOn November 27th at approximately 7:16 am PST, Okta began receiving reports that Okta Verify on iOS was crashing for some users.\r\n\r\nThe issue was constrained to Okta Verify 6.0.1 on iOS 12 (an OS version no longer supported by Okta Verify) and affected at most 0.14% of the Okta Verify user base. Okta Verify 6.0.1 had been released to a small user base on the 26th. Okta immediately advised customers through the service Status page to not upgrade manually to Okta Verify 6.0.1 if running iOS 12. Throughout the incident Okta Verify for iOS 13 and iOS 14, as well as Okta Verify for Android, continued to work as expected.\r\n\r\nFurther investigation determined that an incorrect app metadata change erroneously allowed iOS 12 users to download and install Okta Verify 6.0.1. Okta then initiated a parallel effort to address the issue:\r\n\r\nAs a workaround, on November 28th Okta released Okta Verify 6.0.2 and removed 6.0.1 from the Apple App Store. Metadata in Okta Verify 6.0.2 was appropriately tagged as not supporting iOS 12. Since Okta had removed 6.0.1 from the App Store, users on iOS 12, wanting to download Okta Verify, would be given Okta Verify version 5.9.0, the last release of Okta Verify that was compatible with iOS 12. Okta notified customers through its Status page that users in need of an immediate solution could delete and reinstall the app.\r\n\r\nAs a long term fix, on November 29th Okta released Okta Verify 6.0.4, which reinstated support for iOS 12 and allowed users to upgrade seamlessly.\r\n\r\nOkta is expanding its test matrix to consistently validate that devices running unsupported iOS and Android operating system versions do not receive Okta application upgrades.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRF2UAO"},"Id":"a1P1Y000009QRF2UAO","CreatedDate":"2020-12-15T16:23:32.000+0000","IncidentId__c":"a9C1Y0000008PfEUAU","UpdateLog__c":"Root Cause Analysis:\r\nOn Wednesday December 9th at approximately 9:50 pm (PST) Okta detected an increase in error rates for users in EMEA Cell EU1. Investigation revealed that some requests to the Okta service during this time resulted in 500 errors.\r\n \r\nIt was determined after further analysis that the success rate of requests was highly intermittent during this period with >80% success at some instances in time, and <10% during others.\r\n\r\nThe root cause of this issue was traced to a misconfiguration that caused a resource exhaustion situation within the load balancer tier due to increased request volume. As the service failed, it automatically restarted repeatedly which resulted in the intermittent success spikes during this period.\r\n\r\nAt approximately 10:18 pm (PST) Okta added additional infrastructure to the impacted tier to mitigate impact while addressing the root cause. At 10:24 pm, service was fully restored.\r\n\r\nOkta is rectifying the misconfiguration of the resource limits in the tier, auditing configuration and reviewing resiliency protections and diagnostic tools for the load balancing tier.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRDVUA4"},"Id":"a1P1Y000009QRDVUA4","CreatedDate":"2020-11-26T00:53:16.000+0000","IncidentId__c":"a9C1Y0000008PepUAE","UpdateLog__c":"We are seeing some improvement in processing and have identified the root cause. We are continuing to work on full mitigation. We will provide the next update by 8PM PST.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRj7UAG"},"Id":"a1P1Y000009QRj7UAG","CreatedDate":"2021-02-05T19:10:28.000+0000","IncidentId__c":"a9C1Y0000008PgWUAU","UpdateLog__c":"Root Cause Analysis:\r\n\r\nAt approximately 12:20 pm (PST) on Tuesday February 2nd, a required infrastructure maintenance event caused a partial loss of network connectivity to the LDAP Interface for Okta customers within US Cell 5.\r\n\r\n\r\nDue to mis-prioritization of the monitoring alerts that fired, Okta did not get automatically notified of the impact. At 12:52 pm (PST) a customer reported lack of access to the LDAP Interface and Okta began investigating. Okta isolated the root cause of the issue and restarted service components to restore network connectivity. The LDAP Interface service was fully restored to all customers on US Cell 5 by 4:50pm (PST).\r\n\r\n\r\nImmediately after resolution, Okta implemented updates to internal monitoring and alerting in order to close the identified monitoring gap. As part of an additional internal investigation, Okta has identified further action items to improve the infrastructure maintenance automation as well as monitoring and alerting at additional layers.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRjCUAW"},"Id":"a1P1Y000009QRjCUAW","CreatedDate":"2021-02-05T23:12:18.000+0000","IncidentId__c":"a9C1Y0000008PgbUAE","UpdateLog__c":"Root Cause Analysis:\r\nOn Tuesday February 2nd at approximately 3:46 pm (PST) Okta began receiving customer reports that end users were unable to authenticate into their Microsoft Office 2012 Desktop Clients applications on Microsoft Windows.\r\n\r\nUpon investigation, Okta determined that this failure was caused by an update to the Okta Sign In Widget which had already been deployed to all Preview and Production cells except US Cell 5. Furthermore, any version of Microsoft office client application using Edge 17 (and lower) besides 2012 were also impacted.\r\n\r\nAt 9:21 am (PST) Okta reverted the update of the Sign In Widget to the previous version on all affected cells and the service was fully restored.\r\n\r\nWhile Legacy Edge Browser versions 17 and lower are <a href=\"https://help.okta.com/en/prod/Content/Topics/Miscellaneous/Platforms_Browser_OS_Support.htm\">not officially supported,</a>Okta recognizes that version 17 could still be in use as the embedded browser for some versions of Office Desktop clients. Consequently, Okta is expanding its test matrix to incorporate Office clients with embedded legacy Edge browser version 17.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRjHUAW"},"Id":"a1P1Y000009QRjHUAW","CreatedDate":"2021-02-06T02:31:33.000+0000","IncidentId__c":"a9C1Y0000008PggUAE","UpdateLog__c":"Root Cause Analysis:\r\n\r\nOn Wednesday February 3rd at approximately 5:36 am (PST) Okta began receiving customer reports of Active Directory (AD) imports being delayed. Okta Support worked with Engineering to mitigate the impact by increasing job processing capacity and manually prioritizing impacted jobs while Engineering investigated the root cause of this issue. By 2:05 pm PST this issue was identified to affect all US, EU, and APAC Preview and Production Cells. \r\n\r\n\r\nUpon investigation, Okta determined a code change introduced as part of the 2021.01.2 release update that rolled out to production cells between February 1st and 2nd was the root cause. The code change caused an unintended miscalculation of available job processing capacity for specific job types under specific conditions. Customers with a higher number of import jobs were disproportionately impacted. Throughout the impact window, Customer Support worked with customers and Engineering to optimize communication and our efforts to mitigate customer impact\r\n\r\n\r\nAt 3:54 pm (PST) a fix was submitted for an emergency release which was deployed to all cells between 5:39pm PST and 6:49pm PST. Job processing recovered for all customers by 7:54 pm.\r\n\r\n \r\nOkta is expanding testing in this area to prevent recurrence of this issue.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRjqUAG"},"Id":"a1P1Y000009QRjqUAG","CreatedDate":"2021-02-20T23:27:12.000+0000","IncidentId__c":"a9C1Y0000008PhPUAU","UpdateLog__c":"Resolved: All authentications in Okta cell 5 are now succeeding. We are continuing to look into a remaining cosmetic issue where certain static images (such as logos and background images) aren’t loading properly but this is not impacting functionality. Root cause analysis will be completed within the next 2 business days.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRjvUAG"},"Id":"a1P1Y000009QRjvUAG","CreatedDate":"2021-02-21T20:08:39.000+0000","IncidentId__c":"a9C1Y0000008Ph5UAE","UpdateLog__c":"Root Cause Analysis:\r\nOn Tuesday February 16th at approximately 10:43 am (PST) Okta began receiving customer reports of intermittent errors while using Okta to provision users to the Office365 application. There was no impact to authentication for provisioned users.\r\n\r\nUpon investigation, Okta determined that the protocol leveraged by Okta for provisioning users and making configuration changes to Office 365 was no longer compatible with a new Office 365 version that was being gradually rolled out which resulted in the intermittent errors.\r\n\r\nOn February 17th at 3:45 pm (PST), Okta began to deploy a fix on a cell by cell basis. On February 18th at 2:12 am (PST) the fix was applied to all production cells and service was fully restored.\r\n\r\nOkta is implementing improvements in alerting to further reduce the response time for these types of integration errors as well as exploring the possibility with Microsoft to improve compatibility testing with pre-release versions of Office 365.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P1Y000009QRk0UAG"},"Id":"a1P1Y000009QRk0UAG","CreatedDate":"2021-02-24T05:47:30.000+0000","IncidentId__c":"a9C1Y0000008PhPUAU","UpdateLog__c":"Root Cause Analysis: \r\nAt 12:04pm PST on February 20th, Okta monitoring alerted that static assets were intermittently unavailable in US Cell 5. Upon investigation it was determined that Okta was returning HTTP Response Code 404 (Not Found error) for some static asset requests. At 1:15pm PST the unavailable response rate increased in prevalence and, in some cases depending on the static asset requested, it could impact the ability to sign in to Okta. End user and admin dashboards were also affected. Continued investigation revealed that overly aggressive cache expiration of static assets compounded the problem.\r\n\r\n\r\nBy 2:14pm PST Okta enabled an alternate CDN configuration which mitigated page generation errors. However, some customers were still impacted by missing UI resources such as custom logos, backgrounds, or profile pictures and custom app icons. At 2:59pm PST, Okta made additional changes to load static content not served by the CDN provider. This further reduced customer impact related to custom logos and backgrounds.\r\n\r\n\r\nAt 3:01pm PST the unavailable rate for static assets was mitigated for customers across US Cell 5 and the service was functioning as expected.\r\n\r\n\r\nThe root cause of the service disruption was a load balancer configuration that triggered a service protection response by the block storage service when attempting to retrieve assets. Okta has reconfigured it's load balancers across all cells to avoid a recurrence of the issue. Okta is also developing monitoring and tools to accelerate detection, diagnosis and mitigation of static asset storage related issues.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000FgbdXUAR"},"Id":"a1P2A00000FgbdXUAR","CreatedDate":"2018-03-20T11:30:33.000+0000","IncidentId__c":"a9C2A0000004CSTUA2","UpdateLog__c":"Root Cause Analysis: \r\n\r\nOn March 16th, beginning at 8:23 am PDT, Okta experienced an issue impacting US Cell 6 Administrators' ability to view recently processed user updates in the admin dashboard. The dashboard queries data from secondary read-only databases which were impacted by a replication error. \r\n\r\nThe root cause was traced to an internal database engine error on the primary database which required a consistency check on the secondaries. Addressing the replication failure and resultant latency was delayed because its alert was superseded by alerts for the database engine error. Shortly after the replication latency was escalated by Support, the issue was cleared by firstly re-directing the impacted traffic to the primary databases, and secondly by repairing the replication lag. By 10:22am PDT, Administrator dashboards displayed all current data. Okta is implementing recurrent alerting for replication lags to ensure faster response times going forward.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000Fgbi3UAB"},"Id":"a1P2A00000Fgbi3UAB","CreatedDate":"2018-03-20T22:34:12.000+0000","IncidentId__c":"a9C2A0000004CSnUAM","UpdateLog__c":"Okta continues to investigate an issue with Provisioning Setup and IDP-Initiated Single Sign-On functionality for the following Office 365 Apps: Yammer, CRM and Teams. SP-Initiated Single-Sign-On is not affected. Please contact support if you are having difficulty setting up provisioning and we will assist with implementation of an applicable workaround as we investigate further.<br><br>","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000Fgbi8UAB"},"Id":"a1P2A00000Fgbi8UAB","CreatedDate":"2018-03-20T23:25:38.000+0000","IncidentId__c":"a9C2A0000004CSnUAM","UpdateLog__c":"Okta continues to investigate an issue with Provisioning Setup and IDP-Initiated Single Sign-On functionality for the following Office 365 Apps: Yammer, CRM and Teams. SP-Initiated Single-Sign-On is not affected. Please contact support if you are having difficulty setting up provisioning and we will assist with implementation of an applicable workaround as we investigate further. Another update is planned for 6:25pm PT or sooner as additional information becomes available.<br><br>","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000FgbiDUAR"},"Id":"a1P2A00000FgbiDUAR","CreatedDate":"2018-03-21T01:27:26.000+0000","IncidentId__c":"a9C2A0000004CSnUAM","UpdateLog__c":"Okta continues to investigate an issue with Provisioning Setup and IDP-Initiated Single Sign-On functionality for the following Office 365 Apps: Yammer, CRM and Teams. SP-Initiated Single-Sign-On is not affected. Please contact support if you are having difficulty setting up provisioning and we will assist with implementation of an applicable workaround as we investigate further. Another update is planned for 7:00pm PT or sooner as additional information becomes available.<br><br>","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000FgbiNUAR"},"Id":"a1P2A00000FgbiNUAR","CreatedDate":"2018-03-21T02:25:27.000+0000","IncidentId__c":"a9C2A0000004CSnUAM","UpdateLog__c":"Okta continues to investigate an issue with Provisioning Setup and IDP-Initiated Single Sign-On functionality for the following Office 365 Apps: Yammer, CRM and Teams. SP-Initiated Single-Sign-On is not affected. Please contact support if you are having difficulty setting up provisioning and we will assist with implementation of an applicable workaround as we investigate further. Another update is planned for 9:00pm PT or sooner as additional information becomes available.<br><br>","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000FgbiSUAR"},"Id":"a1P2A00000FgbiSUAR","CreatedDate":"2018-03-21T03:19:22.000+0000","IncidentId__c":"a9C2A0000004CSnUAM","UpdateLog__c":"Okta is currently working to mitigate the issue with IDP-Initiated Single Sign-On functionality for the following Office 365 Apps: Yammer, CRM and Teams. We are exploring a patch with a workaround while we work on a permanent fix. We will continue to provide updates as plans solidify to patch. In addition, please continue to contact support if you are having difficulty setting up O365 provisioning and we will assist with implementation of an applicable workaround for that problem. Another update is planned for 10:30pm PT or sooner as additional information becomes available.<br><br>","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000FgbiXUAR"},"Id":"a1P2A00000FgbiXUAR","CreatedDate":"2018-03-21T05:36:54.000+0000","IncidentId__c":"a9C2A0000004CSnUAM","UpdateLog__c":"Okta is currently working to mitigate the issue with IDP-Initiated Single Sign-On functionality for the following Office 365 Apps: Yammer, CRM and Teams. We are actively working on a patch deployment with a workaround while we work on a permanent fix. We will continue to provide updates as plans solidify to patch. In addition, please continue to contact support if you are having difficulty setting up O365 provisioning and we will assist with implementation of an applicable workaround for that problem. Another update is planned for 12:30 am PT on 03/21/2018 or sooner as additional information becomes available.\r\n<br><br>","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000FgbicUAB"},"Id":"a1P2A00000FgbicUAB","CreatedDate":"2018-03-21T07:33:17.000+0000","IncidentId__c":"a9C2A0000004CSnUAM","UpdateLog__c":"Okta is currently working to mitigate the issue with IDP-Initiated Single Sign-On functionality for the following Office 365 Apps: Yammer, CRM and Teams. We are actively working on deploying a patch to all cells with a workaround while we work on a permanent fix. In addition, please continue to contact support if you are having difficulty setting up O365 provisioning and we will assist with implementation of an applicable workaround for that problem. Another update will be provided as soon all cells have been successfully patched.\r\n<br><br>","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000FgbimUAB"},"Id":"a1P2A00000FgbimUAB","CreatedDate":"2018-03-21T10:43:22.000+0000","IncidentId__c":"a9C2A0000004CSnUAM","UpdateLog__c":"Okta has completed the deployment of a patch to all affected cells with a workaround to address the issue with IDP-Initiated Single Sign-On functionality for the following Office 365 Apps: Yammer, CRM and Teams. SP initiated authentication was unaffected by this issue. We are working on a permanent fix and will provide root cause analysis once it’s available.<br><br>","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000Fgbt6UAB"},"Id":"a1P2A00000Fgbt6UAB","CreatedDate":"2018-03-22T21:42:13.000+0000","IncidentId__c":"a9C2A0000004CSnUAM","UpdateLog__c":"Root Cause Analysis:\r\n\r\nOn 3/20/2018 at 9:50am PDT, Okta identified an issue with IDP-Initiated Single Sign-On functionality for the following Office 365 Apps: Yammer, Teams and some CRM instances. SP initiated authentication was unaffected by this issue. In addition, some customers were unable to complete the Advanced API Access setup for newly created Office 365 application instances. The service disruption was caused by a deactivation of an Okta tenant on Azure Active Directory. Okta deployed workaround to all affected cells from 11:32PM to 4:11am PDT on 3/20. The required AAD tenant was recovered on 3/21 which restored full functionality. In addition, we are reevaluating our internal processes with a focus on third party dependencies required for custom sign on protocols such as the ones used by these applications.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000Fj03vUAB"},"Id":"a1P2A00000Fj03vUAB","CreatedDate":"2017-05-15T13:52:05.000+0000","IncidentId__c":"a9C2A000000blruUAA","UpdateLog__c":"Okta is continuing to investigate the cause of the slow interactive response times in US Cell 2.  Admins may also experience delays in Job processing while we continue our investigation.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000Fj045UAB"},"Id":"a1P2A00000Fj045UAB","CreatedDate":"2017-05-15T14:19:12.000+0000","IncidentId__c":"a9C2A000000blruUAA","UpdateLog__c":"Interactive user experience has returned to normal in US Cell 2.  Admins may continue to experience delays in job processing while we investigate the root cause.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000Fj04ZUAR"},"Id":"a1P2A00000Fj04ZUAR","CreatedDate":"2017-05-15T15:03:33.000+0000","IncidentId__c":"a9C2A000000blruUAA","UpdateLog__c":"RESOLVED: 8:02am PDT: An issue impacting system responsiveness, including interactive performance and job processing in US Cell 2 was addressed.  Additional root cause information will be provided here shortly.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000Fjr2PUAR"},"Id":"a1P2A00000Fjr2PUAR","CreatedDate":"2018-03-30T15:15:50.000+0000","IncidentId__c":"a9C2A000000bmK1UAI","UpdateLog__c":"We are currently working with Microsoft who has acknowledged the issue via incident MO133118. As a workaround, customers can utilize portal.microsoft.com to access the Microsoft Office 365 Portal. More details of Microsoft incident MO133118 below:\r\n<br><br>\r\nhttps://portal.office.com/adminportal/home#/servicehealth. \r\n<br><br>\r\nMO133118 - Unable to access Office 365 portal\r\nStatus:Service degradation\r\nUser impact:Users may be unable to access the Office 365 portal.\r\nLatest message:Title: Unable to access Office 365 portal\r\n\r\nUser Impact: Users may be unable to access the Office 365 portal.\r\n\r\nMore info: Users attempting to access the Office 365 portal via https://portal.microsoftonline.com my receive the following error message: \"This site can't be reached\".\r\n\r\nSome users may be able to access the Office 365 portal via https://portal.office.com.\r\n\r\nCurrent status: We're analysing diagnostic data to isolate the root cause and identify steps to remediate impact.\r\n\r\nScope of impact: This issue could potentially affect any of your users if they are routed through the affected infrastructure.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000Fjr2ZUAR"},"Id":"a1P2A00000Fjr2ZUAR","CreatedDate":"2018-03-30T15:49:50.000+0000","IncidentId__c":"a9C2A000000bmK1UAI","UpdateLog__c":"Okta is monitoring progress with Microsoft on this issue. Recent mitigating steps have been taken to address the issue and we are actively monitoring this incident until fully resolved.\r\n<br><br>","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UoBUAV"},"Id":"a1P2A00000G0UoBUAV","CreatedDate":"2018-03-07T18:20:10.000+0000","IncidentId__c":"a9C2A000000PBikUAG","UpdateLog__c":"We are working to revert the change impacting API Access Management in our US Preview Cell. If you need immediate assistance in resolving the issue, please contact Okta Customer Support.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UoaUAF"},"Id":"a1P2A00000G0UoaUAF","CreatedDate":"2018-03-07T19:04:09.000+0000","IncidentId__c":"a9C2A000000PBikUAG","UpdateLog__c":"The issue impacting API Access Management in US Preview cell has been resolved for the majority of customers. Work is on-going to fully restore the feature for all remaining customer tenants.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UouUAF"},"Id":"a1P2A00000G0UouUAF","CreatedDate":"2018-03-07T19:35:41.000+0000","IncidentId__c":"a9C2A000000PBikUAG","UpdateLog__c":"The issue impacting API Access Management in US Preview cell has been resolved for the majority of customers. Work is on-going to fully restore the feature for all remaining customer tenants. Customers experiencing any issues related to API Access Management should contact Support.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UozUAF"},"Id":"a1P2A00000G0UozUAF","CreatedDate":"2018-03-07T20:13:43.000+0000","IncidentId__c":"a9C2A000000PBikUAG","UpdateLog__c":"The issue impacting API Access Management in US Preview cell has been resolved for the majority of customers. Work is on-going to fully restore the feature for all remaining customer tenants. Customers experiencing any issues related to API Access Management should contact Support.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UpEUAV"},"Id":"a1P2A00000G0UpEUAV","CreatedDate":"2018-03-07T20:28:30.000+0000","IncidentId__c":"a9C2A000000PBikUAG","UpdateLog__c":"An issue impacting API Access Management for all end-users in US Preview Cell was resolved. Additional root cause information will be provided here as soon as it is available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UpTUAV"},"Id":"a1P2A00000G0UpTUAV","CreatedDate":"2018-03-07T20:45:51.000+0000","IncidentId__c":"a9C2A000000PBikUAG","UpdateLog__c":"Okta continues to address issues with the API Access Management feature. Customers experiencing issues are advised to contact support.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UpsUAF"},"Id":"a1P2A00000G0UpsUAF","CreatedDate":"2018-03-07T21:25:07.000+0000","IncidentId__c":"a9C2A000000PBikUAG","UpdateLog__c":"The issue impacting API Access Management in US Preview cell has been resolved for the majority of customers. Work is on-going to fully restore the feature for all remaining customer tenants. Customers experiencing any issues related to API Access Management should contact Support.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0Uq2UAF"},"Id":"a1P2A00000G0Uq2UAF","CreatedDate":"2018-03-07T21:50:15.000+0000","IncidentId__c":"a9C2A000000PBikUAG","UpdateLog__c":"An issue impacting API Access Management for all end-users in US Preview Cell was resolved. Additional root cause information will be provided here as soon as it is available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UqlUAF"},"Id":"a1P2A00000G0UqlUAF","CreatedDate":"2018-03-08T01:50:29.000+0000","IncidentId__c":"a9C2A000000PBipUAG","UpdateLog__c":"An issue impacting system responsiveness in US Cell 6 is being investigated.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UqvUAF"},"Id":"a1P2A00000G0UqvUAF","CreatedDate":"2018-03-08T02:15:09.000+0000","IncidentId__c":"a9C2A000000PBipUAG","UpdateLog__c":"An issue impacting system availability and authentication in US Cell 6 is being mitigated.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0Ur0UAF"},"Id":"a1P2A00000G0Ur0UAF","CreatedDate":"2018-03-08T02:45:44.000+0000","IncidentId__c":"a9C2A000000PBipUAG","UpdateLog__c":"An issue impacting system availability in US Cell 6 has been resolved. Additional root cause information will be provided here as soon as it is available","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UyNUAV"},"Id":"a1P2A00000G0UyNUAV","CreatedDate":"2018-03-09T23:40:54.000+0000","IncidentId__c":"a9C2A000000PBipUAG","UpdateLog__c":"Root Cause Analysis:\r\n\r\nOn March 7th, beginning at 5:15pm PST, Okta experienced a service disruption in US Cell 6. End-User, Admin and API requests within Okta tenants within US Cell 6 were slow, unresponsive, or encountered HTTP 5xx errors.\r\n\r\nThe disruption was the result of a memory issue which impacted the cell's Redis cluster which is used to cache application data. The root cause of the memory issue was traced to a combination of high memory usage within the Redis cluster, a mis-configured application server which increased the volume of caching above expected levels, and sub-optimal Redis configuration settings. Because both the primary and secondary Redis tiers were equally impacted by the increased memory demands, the cluster was unable to recover. To stabilize the Redis cluster, additional capacity was added and application servers were recycled to clear hung connections and re-balance load. By 6:08 pm PST, the cell had returned to normal performance levels.\r\n\r\nOkta has completed work to up-size our Redis caching clusters in US Cell 6 and will deploy the same mitigating changes across all remaining cells. Additionally, Okta will conduct exercises to replay this failure sequence within a test environment to determine and subsequently deploy the optimal settings for Redis memory management, client connection configuration and health-check thresholds. Additionally, new tooling is being investigated to allow us to flush Redis cache should such a mitigation step be needed in the future.\r\n<br><br> \r\nReview the root cause analysis [here](https://support.okta.com/help/Documentation/Knowledge_Article/Root-Cause-Analysis-Service-Disruption-03-07-2017).","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P2A00000G0UymUAF"},"Id":"a1P2A00000G0UymUAF","CreatedDate":"2018-03-10T00:15:52.000+0000","IncidentId__c":"a9C2A000000PBikUAG","UpdateLog__c":"Root Cause Analysis:\r\n\r\nOn March 7th, beginning at 9:35am PST, Okta experienced a feature disruption to API Access Management in the US Preview cell. The disruption was caused while Okta was working to correct the assignment of the API Access Management feature to certain tenants and mistakenly removed the feature from orgs authorized to have it.\r\n\r\nAfter the feature was inadvertently disabled, Okta began work to bulk re-assign the feature to all entitled tenants. However, due the complexity of the process of enabling the API Access Management feature, it took Okta longer than usual to restore the feature to entitled tenants. During the incident, while bulk enablement was processing, Okta Support and Engineering manually enabled the feature for all customers who reported the issue or were detected as being impacted through Okta’s monitoring tools.\r\n\r\nBy 10:30 AM PST, the majority of the impacted tenants had the API Access Management feature reassigned at 1:50 PM PST all entitled customers in the US Preview cell had the feature restored.\r\n\r\nTo prevent recurrence, Okta is adding additional oversight to feature deployment processes and will be hardening our management software to reduce risk of data input errors.\r\n\r\n<br><br> \r\nReview the root cause analysis [here](https://support.okta.com/help/Documentation/Knowledge_Article/Root-Cause-Analysis-API-Access-Management-Service-Disruption-03072018).","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKj0EAH"},"Id":"a1P4z00000BAKj0EAH","CreatedDate":"2021-09-01T18:03:17.000+0000","IncidentId__c":"a9C4z000000wk37EAA","UpdateLog__c":"We sincerely apologize for any impact this incident has caused to you and your business and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident.\r\n\r\n\r\n\r\nDetection and Impact:\r\n\r\nAt 8:18PM (PT) on August 28th, 2021 Okta was alerted to instance failures within US Cell 5 in our database tier. The automated response switches application servers into read only mode to continue serving primary use cases. Customers attempting to reach US Cell 5 while in read only mode experienced failures for some post and put requests.\r\n\r\n\r\n\r\nDuring the established procedure to address the instance failures and recover from read only mode, applications servers were set to an incorrect state in which they were not able to reach the primary and read-only database tiers at 8:35PM. As a result, the application servers switched into an offline mode at 8:39PM and stopped serving requests.\r\n\r\n\r\n\r\nFrom 8:39PM to 9:07PM Okta worked to mitigate and recover the application servers. During this period, the success rate was about 40% for customers accessing US Cell 5 due to the applications fluctuating between an available and unavailable state. At 9:07PM application servers were set to the correct active state with full access to the primary and read-only database tiers. At 9:09PM service was restored in US Cell 5.\r\n\r\n\r\n\r\nRoot Cause Summary:\r\n\r\nThe root cause of the incident is the instance failures in US Cell 5. The service behaved as designed for this failure case and went into read-only mode.\r\n\r\n\r\n\r\nIn response to the failure, the operation to restore the service to active mode, the application servers were set into the wrong mode due to a manual error by the engineer executing a recovery procedure. \r\n\r\n\r\n\r\nPreventive Actions:\r\n\r\nAs a result of this issue, Okta is working to augment the established procedure to address instance failures to prevent setting the application servers into a state that prevents them from reaching primary and read-only data sources.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1waDEAR"},"Id":"a1P4z00000B1waDEAR","CreatedDate":"2021-03-09T18:59:52.000+0000","IncidentId__c":"a9C4z000000oLkPEAU","UpdateLog__c":"Root Cause Analysis: \r\nOn March 7 at 8:20am PST Okta detected intermittent network connectivity to and from OK8 Cell (APAC). Investigation revealed that some access requests to the Okta service during this time could not be successfully completed.\r\n\r\nAnalysis determined that an external internet provider issue caused the connection errors. Further investigation revealed that the success rate of requests was highly intermittent during this period and also dependent on the location of the Okta user. The majority of requests within Australia were successful, while connection attempts outside of Australia had higher failure rates. The highest concentration of unsuccessful attempts was observed between 9:20am PST to 9:30am PST.\r\n\r\nAt approximately 9:35am PST, our service provider mitigated the internet connectivity issue by redirecting international traffic to Australia through an alternate provider and by 11:00am PST the service was operating as expected.\r\n\r\nGiven the lack of information regarding the root cause and scope of the service provider issue and how that might impact access of Australian users to our Disaster Recovery cell in Singapore, Okta elected not to reroute traffic to the Disaster Recovery cell \r\n\r\nOkta is working closely with the provider to investigate improved mitigations to improve response to these types of incidences.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1wvQEAR"},"Id":"a1P4z00000B1wvQEAR","CreatedDate":"2021-03-16T20:27:46.000+0000","IncidentId__c":"a9C4z000000oLkeEAE","UpdateLog__c":"Resolved: Microsoft has reported that Azure AD service is restored. All Okta Services related to Microsoft Office 365 and Azure are working as expected with the exception of some connections to Azure Active Directory Graph API for Okta Provisioning service which we expect to be slower to recover. We will continue to monitor residual effects for a limited number of customers. We are working with the service provider to investigate the root cause of this service disruption and will post the RCA to [status.okta.com](http://status.okta.com.com) within 48 hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1xA7EAJ"},"Id":"a1P4z00000B1xA7EAJ","CreatedDate":"2021-03-22T16:26:23.000+0000","IncidentId__c":"a9C4z000000oLkeEAE","UpdateLog__c":"Root Cause Analysis:\r\nOn March 15th 2021 at 12:43pm PDT, Okta received automated alerts that Microsoft Office 365 endpoints were returning errors. Upon investigation it was revealed that several Microsoft services that used Azure Active Directory (Azure AD) like Office 365, Dynamics 365, Azure, Teams, Exchange Online, Intune and Yammer were affected by the outage as per status.office.com. Okta Provisioning and SSO services were affected as a result of this outage across all cells.\r\n\r\nOn March 16th, at 2:25am PDT, Microsoft declared the incident resolved. At approximately 2:30pm PDT, Okta started to see improvements in Single Sign-On (SSO) and the Provisioning services. Okta observed a gradual downward trend in errors from the Azure AD endpoints. At 4:45pm PST, Okta SSO services were restored across all cells. \r\n\r\nOn March 16th 2021 at 1:22pm PDT Okta confirmed that the Azure AD service to Okta Provisioning and SSO services was restored for the majority of our customers. However, residual impacts to provisioning were observed for some customers as Microsoft propagates changes across Microsoft’s services.\r\n\r\nWhile the majority of Okta’s customers have fully recovered, as part of Okta’s on-going commitment to our customers, we are assisting our customers in working w/ Microsoft to address any residual issues as a result of the Microsoft outage.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKmJEAX"},"Id":"a1P4z00000BAKmJEAX","CreatedDate":"2021-10-29T23:35:51.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Root Cause Analysis\r\n\r\nWe sincerely apologize for any impact this incident has caused to you and your business and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident.\r\n\r\nOn October 27th at 4:11AM PDT, customers began reporting intermittent delays for MFA requests that used AT&T as the SMS provider for US based numbers. At 6:23AM PDT Okta’s systems showed a return to normal SMS delivery. Okta continued to closely monitor the situation and confirmed at 2:55PM PDT that the carrier’s mitigation efforts were completed.\r\n\r\nDetection and Impact:\r\nIn all cells, customers who used the AT&T carrier were impacted in their SMS MFA requests. Customers did not receive, or experienced very long delivery delays for, their one time passcode requests through SMS. Okta confirmed with our downstream telecommunications providers that this was an issue with AT&T.\r\n\r\nRoot Cause Summary:\r\nAT&T experienced network issues across the US. This caused SMS delivery delays and failures for the subscribers of their service.\r\n\r\nPreventive Actions:\r\nThe root cause of the issue occurred within the telecommunication infrastructure rather than with one of our downstream telecommunications providers. Okta has added additional monitoring and alerting to more rapidly detect such issues in the future; enabling us to validate the issue with our downstream providers and react accordingly (e.g. post to Okta Status | System Status ).","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKiCEAX"},"Id":"a1P4z00000BAKiCEAX","CreatedDate":"2021-08-29T04:21:52.000+0000","IncidentId__c":"a9C4z000000wk37EAA","UpdateLog__c":"Okta has resolved the issue in OK5 and the service is restored. Additional root cause information will provided within 2 business days.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKibEAH"},"Id":"a1P4z00000BAKibEAH","CreatedDate":"2021-08-31T21:28:58.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We continue to mitigate issues observed in US Cells 5, 7, 12 and 14, and Preview 3. Error rates have reduced for users and admins. We are actively engaged and will share updates as soon as we have more information.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKigEAH"},"Id":"a1P4z00000BAKigEAH","CreatedDate":"2021-08-31T22:15:14.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We continue to mitigate issues observed in US Cells 5, 7, 12, 14 and Preview 3. The error rates have significantly subsided and we are seeing recovery across all the affected cells. We will share updates as soon as we have more information.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKivEAH"},"Id":"a1P4z00000BAKivEAH","CreatedDate":"2021-08-31T23:15:55.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We have mitigated all observed errors and issues in the US-West region affecting US cells 5, 7, 12, 14, EU 9, and Preview 3. Error rates have completely subsided and the service has returned to normal status, queued events are progressing and should complete in the next few hours. Additional root cause information will provided within 2 business days.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKiqEAH"},"Id":"a1P4z00000BAKiqEAH","CreatedDate":"2021-08-31T22:53:34.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We have mitigated all observed errors and issues in the US-West region affecting US cells 5, 7, 12, 14, and Preview 3. Error rates have completely subsided and the service has returned to normal status, queued events are progressing and should complete in the next few hours.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKn2EAH"},"Id":"a1P4z00000BAKn2EAH","CreatedDate":"2021-11-09T00:15:07.000+0000","IncidentId__c":"a9C4z000000wk5cEAA","UpdateLog__c":"Okta is seeing sustained successes in Workflow Console, flow invocation, and all services are running without any issues. The workflows system is currently processing a backlog so customers may experience higher than normal latency on their hook requests. We are monitoring and will provide an update on the latency in 30 minutes","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKiREAX"},"Id":"a1P4z00000BAKiREAX","CreatedDate":"2021-08-31T20:25:06.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We are currently mitigating error rates observed in US Cells 5, 7, 12, and 14, and Preview 3. Error rates are improving and we remain actively engaged on the issue. Further status updates will be provided as soon as we have more information.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKmsEAH"},"Id":"a1P4z00000BAKmsEAH","CreatedDate":"2021-11-08T22:57:52.000+0000","IncidentId__c":"a9C4z000000wk5cEAA","UpdateLog__c":"Okta is experiencing an issue where admins may be experiencing errors while utilizing Workflow designer and invoking flows. A delay in event hook delivery has been identified and is being addressed. Scheduled flows are not affected and continue to run without issue. Engineering is addressing the issue. We will provide an update within the next 30 minutes.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKmxEAH"},"Id":"a1P4z00000BAKmxEAH","CreatedDate":"2021-11-08T23:29:22.000+0000","IncidentId__c":"a9C4z000000wk5cEAA","UpdateLog__c":"Okta is experiencing an issue where admins may be experiencing errors while utilizing Workflow designer and invoking flows. A delay in event hook delivery has been identified and is being addressed. Scheduled flows are not affected and continue to run without issue. Engineering is addressing the issue. We will provide an update within the next 30 minutes.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKiWEAX"},"Id":"a1P4z00000BAKiWEAX","CreatedDate":"2021-08-31T21:03:55.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We continue to mitigate issues observed in US Cells 5, 7, 12 and 14, and Preview 3. Error rates have reduced for users and admins. We are actively engaged and will share updates as soon as we have more information.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKlfEAH"},"Id":"a1P4z00000BAKlfEAH","CreatedDate":"2021-10-27T15:41:25.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Okta continues to monitor the current situation with our partners regarding the AT&T outage. Our monitoring shows a return to normal conditions with MFA for customers that utilize SMS. However, we have yet to receive confirmation from our providers that the issue is fully resolved. Our next update will be in one hour or sooner if additional information becomes available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKiHEAX"},"Id":"a1P4z00000BAKiHEAX","CreatedDate":"2021-08-31T19:21:19.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We continue to investigate the increased error rates in US Cells 5, 7, 12, and 14, and Preview 3. We are actively investigating the issue and will provide a status update as soon as we have further information.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1xSZEAZ"},"Id":"a1P4z00000B1xSZEAZ","CreatedDate":"2021-04-05T13:11:36.000+0000","IncidentId__c":"a9C4z000000oLkyEAE","UpdateLog__c":"Root Cause Analysis:\r\nOn April 1st 2021 at 2:43pm PDT, Okta began receiving automated alerts that Microsoft Office 365 endpoints were returning errors. \r\n\r\nUpon investigation Okta quickly discovered that a DNS issue reported by Microsoft (status.office.com) impacted access to several Microsoft 365 and Azure services such as Office 365, Dynamics 365, Azure, Teams, Exchange Online, Intune and Yammer. Okta Provisioning and SSO integrations with Microsoft services were affected across all cells by this issue.\r\n\r\nAt approximately 3:36 pm Microsoft reported that they had rerouted DNS traffic for their services and after 3:59pm PDT Okta observed no further errors.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1wteEAB"},"Id":"a1P4z00000B1wteEAB","CreatedDate":"2021-03-16T15:47:15.000+0000","IncidentId__c":"a9C4z000000oLkeEAE","UpdateLog__c":"Microsoft has reported that the Azure AD service has been restored. All Okta Services related to Microsoft Office 365 and Azure are working as expected with the exception of some connections to Azure Active Directory Graph API for Okta Provisioning service will be slower to recover.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1wrOEAR"},"Id":"a1P4z00000B1wrOEAR","CreatedDate":"2021-03-15T21:32:35.000+0000","IncidentId__c":"a9C4z000000oLkeEAE","UpdateLog__c":"Microsoft is reporting a system wide outage impacting Azure and Office 365 applications. Okta services impacted due to the Microsoft outage include: Provisioning to Azure AD and some SSO. Okta continues to monitor the situation. We encourage you to go to the latest updates on Microsoft's Status at [status.okta.com](http://status.office.com).","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1wrsEAB"},"Id":"a1P4z00000B1wrsEAB","CreatedDate":"2021-03-15T22:44:56.000+0000","IncidentId__c":"a9C4z000000oLkeEAE","UpdateLog__c":"Microsoft continues to report a system wide outage impacting Azure and Office 365 applications. Okta services impacted due to the Microsoft outage include: Provisioning to Azure AD. Okta continues to monitor the situation. We encourage you to go to the latest updates on Microsoft's Status at [status.okta.com](http://status.office.com).","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKm4EAH"},"Id":"a1P4z00000BAKm4EAH","CreatedDate":"2021-10-27T20:40:13.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Our monitoring shows a return to normal conditions with MFA for customers that utilize SMS with AT&T. However, we have yet to receive confirmation from our providers that the issue is fully resolved. Our next update will be in one hour or sooner if additional information becomes available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKmnEAH"},"Id":"a1P4z00000BAKmnEAH","CreatedDate":"2021-11-08T22:19:12.000+0000","IncidentId__c":"a9C4z000000wk5cEAA","UpdateLog__c":"Okta is experiencing an issue where admins may be experiencing errors while utilizing Workflow Console and invoking flows. A delay in event hook delivery has been identified and is being addressed. Scheduled flows are not affected and continue to run without issue. Engineering is currently investigating the issue. We will provide an update within the next 30 minutes.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKkhEAH"},"Id":"a1P4z00000BAKkhEAH","CreatedDate":"2021-10-15T03:00:30.000+0000","IncidentId__c":"a9C4z000000wk4eEAA","UpdateLog__c":"Mitigation steps have been applied.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKlzEAH"},"Id":"a1P4z00000BAKlzEAH","CreatedDate":"2021-10-27T19:40:46.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Our monitoring shows a return to normal conditions with MFA for customers that utilize SMS with AT&T. However, we have yet to receive confirmation from our providers that the issue is fully resolved. Our next update will be in one hour or sooner if additional information becomes available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKkmEAH"},"Id":"a1P4z00000BAKkmEAH","CreatedDate":"2021-10-15T03:09:45.000+0000","IncidentId__c":"a9C4z000000wk4eEAA","UpdateLog__c":"Mitigated the issue and we are monitoring it.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKhiEAH"},"Id":"a1P4z00000BAKhiEAH","CreatedDate":"2021-08-12T02:31:57.000+0000","IncidentId__c":"a9C4z000000wk2sEAA","UpdateLog__c":"Root Cause Analysis:\r\nAt 2:33pm PST on August 9th 2021, Okta’s monitoring alerted Engineering to increased error rates in US Cell 10. Investigation revealed that a planned maintenance was incorrectly applied to a subset of the load balancers in US Cell 10. \r\n\r\n\r\nAt 2:39pm Okta initiated a rollback of the maintenance operation in US Cell 10 and at 2:41p PST error rates started to decrease. As a result of this roll-back, error rates returned to pre-incident levels at 2:43pm. \r\n \r\n\r\nThe planned manual maintenance operation replaced virtual firewall rules to disable connection tracking at the outer edge of the service. During the creation of the new virtual firewall rule the permissive ingress part of the rule was mistakenly omitted, thereby inhibiting requests from reaching the next layer in the infrastructure in US Cell 10. As Okta maintains a restrictive firewall policy that defaults to denying access unless expressly allowed by a rule, the rule replacement further restricted inbound traffic to hosts and caused traffic to be blocked. This maintenance was not applied to Custom Domain, and traffic to Custom Domains was not impacted during this incident in US Cell 10. \r\n\r\n\r\nWe sincerely apologize for the impact to affected customers and are working to improve the Okta service. To prevent recurrence, Okta has implemented a moratorium on maintenance of virtual firewall rules until additional manual process checks and improvements in automation are incorporated to ensure such maintenance is done without customer impact.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKlkEAH"},"Id":"a1P4z00000BAKlkEAH","CreatedDate":"2021-10-27T16:41:17.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Our monitoring shows a return to normal conditions with MFA for customers that utilize SMS with AT&T. However, we have yet to receive confirmation from our providers that the issue is fully resolved. Our next update will be in one hour or sooner if additional information becomes available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKluEAH"},"Id":"a1P4z00000BAKluEAH","CreatedDate":"2021-10-27T18:42:19.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Our monitoring shows a return to normal conditions with MFA for customers that utilize SMS with AT&T. However, we have yet to receive confirmation from our providers that the issue is fully resolved. Our next update will be in one hour or sooner if additional information becomes available.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000B1xW7EAJ"},"Id":"a1P4z00000B1xW7EAJ","CreatedDate":"2021-06-15T21:42:06.000+0000","IncidentId__c":"a9C4z000000oLmzEAE","UpdateLog__c":"Root Cause Analysis:\r\n\r\nProduct: Okta Workflows\r\n\r\nOn June 11, 2021 at approximately 10:14AM PDT, Okta detected a higher volume of Workflows execution requests on US Cells 1, 2, 3, 4, 6, 7 and 11 impacting the workflows service. Shortly after, Okta monitoring began to report an increase in error rates. At the same time, Okta began receiving reports from customers who were experiencing timeout errors and they were impacted in the following ways:\r\n\r\n- Workflow API calls did not complete execution and timed out\r\n\r\n- Workflows Console Folder view errors/timeouts were experienced until 11:35AM\r\n\r\n- Workflows Designer Test flow timeout errors were experienced until 12:16PM\r\n \r\nUpon investigation, Okta identified a workflow that was generating a large incoming workload and took immediate action to isolate it. The system load generated by this workload was not automatically distributed across nodes in the cluster as expected. As a result the incoming requests continued to accumulate which degraded overall workflows performance. The cluster eventually became unresponsive. In addition, the health checks between redundant nodes did not report any infrastructure issues. This caused further delays in isolating and remediating the issue.\r\n \r\nOnce Okta isolated the problem to the unresponsive node, we immediately performed a restart of that node at 12:00PM and service was largely restored at approximately 12:20PM PDT and was fully restored at 12:45PM PDT.\r\n \r\nTo mitigate the impact to workflows for similar future events, Okta continues to invest in system automation to gracefully handle large workloads. Okta is also implementing additional monitoring and alerting to further reduce the time needed for issue identification and resolution.","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKkwEAH"},"Id":"a1P4z00000BAKkwEAH","CreatedDate":"2021-10-19T02:55:53.000+0000","IncidentId__c":"a9C4z000000wk4eEAA","UpdateLog__c":"We sincerely apologize for any impact this incident has caused to you and your business and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident.\r\nDetection and Impact\r\nOn October 14th at approximately 2:30PM PDT, Okta received customer reports of intermittent SSL certificate warnings for client requests to the Okta service. Investigation revealed that this issue impacted custom URL domain users on US Cell 10. In addition, this issue was observed to occur intermittently during this period with the overall success rate of greater than 85% for users. Overall response time and Admin requests were not impacted during this period, as admin requests use the org subdomain instead of the custom domains. Retry attempts would have likely succeeded. \r\n \r\nRoot Cause Summary:\r\nOkta determined that the root cause was a maintenance operation to add router capacity in US Cell 10. During the execution of this operation, a non custom domain router resource was added to the target group which resulted in errors being returned when attempting to establish TLS connections for custom domain requests serviced by this router. The impact was intermittent because requests are handled by a pool of routers, and only requests received by the affected member of the US Cell 10 router pool were impacted. \r\n\r\nRemediation Steps:\r\nAt 7:43PM PDT, Okta began to address the impact by removing the affected router from the Custom Domain Target Group. Following resumption of normal operation, service resumed at 7:44PM PDT.\r\n \r\nPreventative Actions: \r\nOkta has already added additional checks to the automation tool involved to prevent this router mis-assignment. Okta is also improving the process for deploying and verifying SSL certificates as well as adding detection measures to prevent occurrence of this or similar issues in the future.\r\n \r\nDuration (# of minutes): 383","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v53.0/sobjects/IncidentUpdate__c/a1P4z00000BAKlpEAH"},"Id":"a1P4z00000BAKlpEAH","CreatedDate":"2021-10-27T17:41:12.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Our monitoring shows a return to normal conditions with MFA for customers that utilize SMS with AT&T. However, we have yet to receive confirmation from our providers that the issue is fully resolved. Our next update will be in one hour or sooner if additional information becomes available.","CurrencyIsoCode":"USD"}]