{
  "page": {
    "id": "lhlms7kbdd5n",
    "name": "Teleport",
    "url": "https://status.goteleport.com"
  },
  "status": {
    "indicator": "none",
    "description": "All Systems Operational"
  },
  "components": [{"status":"operational","name":"Downloads (cdn.teleport.dev)","created_at":"2019-11-25T15:23:48.491Z","updated_at":"2024-07-04T00:33:37.735Z","position":3,"description":"Our file server for downloading releases","showcase":false,"start_date":null,"id":"s0mms0j5qqv9","group_id":null,"page_id":"lhlms7kbdd5n"},{"status":"operational","name":"GitHub Git Operations","created_at":"2020-05-29T14:37:22.494Z","updated_at":"2026-11-04T11:34:27.268Z","position":6,"description":null,"showcase":false,"start_date":null,"id":"jb1726g4tqy8","group_id":null,"page_id":"lhlms7kbdd5n"},{"status":"operational","name":"AWS Elastic Container Registry","created_at":"2024-02-04T22:57:00.382Z","updated_at":"2025-10-20T21:45:19.151Z","position":7,"description":"https://gallery.ecr.aws/gravitational","showcase":false,"start_date":null,"id":"v42dkdydlq94","group_id":null,"page_id":"lhlms7kbdd5n"}],
  "incidents": [{"name":"Critical security patch available, please update immediately.","status":"resolved","created_at":"2025-06-16T16:04:32.000Z","updated_at":"2025-07-01T00:23:06.350Z","monitoring_at":"2025-06-16T16:04:32.000Z","resolved_at":"2025-07-01T00:23:06.332Z","impact":"critical","shortlink":"https://stspg.io/0cdv8r9gsf3g","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"critical","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2025-06-16T16:04:32.000Z","reminder_intervals":null,"id":"srppdb02swhq","page_id":"lhlms7kbdd5n","incident_updates":[{"status":"resolved","body":"We have published the RCA and you can view it in our \u003ca href=\"https://trust.goteleport.com/resources?s=32t147ja8aawd6px7irxat\u0026name=cve-2025-49825-rca\"\u003eTrust Center\u003c/a\u003e","created_at":"2025-07-01T00:23:06.332Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2025-07-01T00:23:06.332Z","display_at":"2025-07-01T00:23:06.332Z","deliver_notifications":false,"tweet_id":null,"id":"75t3nfrmm0jp","incident_id":"srppdb02swhq","custom_tweet":null,"affected_components":null},{"status":"monitoring","body":"\u003cb\u003eThis is a limited version of a notification we have sent to customers. If you are a customer, refer to the original communication and FAQ  we sent you for more details.\u003c/b\u003e\n\nTeleport security engineers have identified a critical security vulnerability that could allow remote authentication bypass of Teleport .  \n\nTeleport Cloud Infrastructure and CI/CD build, test, and release infrastructure aren’t affected. \n\nFor the full mitigation, we \u003cb\u003erequire you\u003c/b\u003e to upgrade both Proxy and Teleport agents. We also strongly recommend updating clients to the released patch versions as a precaution.\n\n\u003cb\u003eFixed in versions: 17.5.2, 16.5.12, 15.5.3, 14.4.1, 13.4.27, 12.4.35.\u003c/b\u003e\n\nThese patches are available \u003cb\u003eonly\u003c/b\u003e on the \u003ca href=\"https://goteleport.com/docs/installation/\"\u003eofficial Teleport distribution channels\u003c/a\u003e. \n\nWe are designating these versions as Critical Security Exception Versions.  \n\n\u003ci\u003eFor these specific patch versions of Teleport Community Edition, we are removing Community Edition restrictions on employee count or revenue thresholds, as long as you apply the patch within thirty (30) days of its official release.\nPlease read the full text of the \u003ca href=\"https://github.com/gravitational/teleport/blob/master/build.assets/LICENSE-community\"\u003eupdated Teleport Community Edition license\u003c/a\u003e for details.\u003c/i\u003e\n\n\u003ca href=\"https://github.com/gravitational/teleport/security/advisories/GHSA-8cqv-pj7f-pwpc\"\u003eAdvisory Details\u003c/a\u003e","created_at":"2025-06-16T16:04:32.884Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2025-06-16T16:06:01.859Z","display_at":"2025-06-16T16:04:32.000Z","deliver_notifications":false,"tweet_id":null,"id":"xt0tvxx8sxsc","incident_id":"srppdb02swhq","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Teleport v17.3.0 RPM is no longer accessible, please use v17.3.1","status":"resolved","created_at":"2025-03-04T00:07:04.213Z","updated_at":"2025-03-04T04:00:56.010Z","monitoring_at":"2025-03-04T00:07:04.257Z","resolved_at":"2025-03-04T04:00:55.995Z","impact":"none","shortlink":"https://stspg.io/1vxk95l2fk5w","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2025-03-04T00:07:04.208Z","reminder_intervals":null,"id":"p8kl6rq2kjzx","page_id":"lhlms7kbdd5n","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2025-03-04T04:00:55.995Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2025-03-04T04:00:55.995Z","display_at":"2025-03-04T04:00:55.995Z","deliver_notifications":false,"tweet_id":null,"id":"bzf3fmv4dnhl","incident_id":"p8kl6rq2kjzx","custom_tweet":null,"affected_components":[{"code":"s0mms0j5qqv9","name":"Downloads (cdn.teleport.dev)","old_status":"operational","new_status":"operational"}]},{"status":"monitoring","body":"A bug in the Teleport v17.3.0 RPM was discovered that could lead to Teleport services stopping if the system is rebooted after upgrading from an earlier version. The Teleport v17.3.0 RPM is no longer accessible, and Teleport v17.3.1 has been released. If you installed Teleport v17.3.0 from an RPM, we recommend updating to v17.3.1 before your system is rebooted to ensure continued availability. Details: https://github.com/gravitational/teleport/pull/52703","created_at":"2025-03-04T00:07:04.257Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2025-03-04T00:07:04.257Z","display_at":"2025-03-04T00:07:04.257Z","deliver_notifications":true,"tweet_id":null,"id":"bjdy2g5f099k","incident_id":"p8kl6rq2kjzx","custom_tweet":null,"affected_components":[{"code":"s0mms0j5qqv9","name":"Downloads (cdn.teleport.dev)","old_status":"operational","new_status":"operational"}]}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Downloads (cdn.teleport.dev)","created_at":"2019-11-25T15:23:48.491Z","updated_at":"2024-07-04T00:33:37.735Z","position":3,"description":"Our file server for downloading releases","showcase":false,"start_date":null,"id":"s0mms0j5qqv9","page_id":"lhlms7kbdd5n","group_id":null}]},{"name":"Auth Server configuration changes for select tenants","status":"completed","created_at":"2024-01-02T12:21:38.503Z","updated_at":"2024-01-04T14:00:14.830Z","monitoring_at":null,"resolved_at":"2024-01-04T14:00:14.806Z","impact":"maintenance","shortlink":"https://stspg.io/9g2x1cjhp6vk","scheduled_for":"2024-01-04T13:00:00.000Z","scheduled_until":"2024-01-04T14:00:00.000Z","scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":true,"scheduled_auto_completed":true,"metadata":{},"started_at":"2024-01-02T12:21:38.498Z","reminder_intervals":null,"id":"2ljf2d6tlbv3","page_id":"lhlms7kbdd5n","incident_updates":[{"status":"completed","body":"The scheduled maintenance has been completed.","created_at":"2024-01-04T14:00:14.806Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2024-01-04T14:00:14.806Z","display_at":"2024-01-04T14:00:14.806Z","deliver_notifications":false,"tweet_id":null,"id":"7rz0nyl83q65","incident_id":"2ljf2d6tlbv3","custom_tweet":null,"affected_components":null},{"status":"in_progress","body":"Scheduled maintenance is currently in progress. We will provide updates as necessary.","created_at":"2024-01-04T13:00:25.478Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2024-01-04T13:00:25.478Z","display_at":"2024-01-04T13:00:25.478Z","deliver_notifications":false,"tweet_id":null,"id":"b14pqfyhp0nd","incident_id":"2ljf2d6tlbv3","custom_tweet":null,"affected_components":null},{"status":"scheduled","body":"We are applying configuration changes to Teleport Auth Servers to a small number of Enterprise tenants during this time. No customer impact expected.","created_at":"2024-01-02T12:21:38.551Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2024-01-02T12:21:38.551Z","display_at":"2024-01-02T12:21:38.551Z","deliver_notifications":false,"tweet_id":null,"id":"pcxndz072175","incident_id":"2ljf2d6tlbv3","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Auth Server configuration changes for select tenants","status":"completed","created_at":"2023-12-28T12:03:18.505Z","updated_at":"2023-12-29T14:01:42.502Z","monitoring_at":null,"resolved_at":"2023-12-29T14:01:42.474Z","impact":"maintenance","shortlink":"https://stspg.io/6w0wktm79l2t","scheduled_for":"2023-12-29T13:00:00.000Z","scheduled_until":"2023-12-29T14:00:00.000Z","scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":true,"scheduled_auto_completed":true,"metadata":{},"started_at":"2023-12-28T12:03:18.501Z","reminder_intervals":null,"id":"8b8yvwkdvm72","page_id":"lhlms7kbdd5n","incident_updates":[{"status":"completed","body":"The scheduled maintenance has been completed.","created_at":"2023-12-29T14:01:42.474Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2023-12-29T14:01:42.474Z","display_at":"2023-12-29T14:01:42.474Z","deliver_notifications":false,"tweet_id":null,"id":"sfm71tc2rh7h","incident_id":"8b8yvwkdvm72","custom_tweet":null,"affected_components":null},{"status":"in_progress","body":"Scheduled maintenance is currently in progress. We will provide updates as necessary.","created_at":"2023-12-29T13:00:13.823Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2023-12-29T13:00:13.823Z","display_at":"2023-12-29T13:00:13.823Z","deliver_notifications":false,"tweet_id":null,"id":"tc71ztcm16yn","incident_id":"8b8yvwkdvm72","custom_tweet":null,"affected_components":null},{"status":"scheduled","body":"We are applying configuration changes to Teleport Auth Servers to a small number of Enterprise tenants during this time. No customer impact expected.","created_at":"2023-12-28T12:03:18.537Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2023-12-28T12:03:18.537Z","display_at":"2023-12-28T12:03:18.537Z","deliver_notifications":false,"tweet_id":null,"id":"7d9fcgjm88q8","incident_id":"8b8yvwkdvm72","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Invalid License Warning","status":"resolved","created_at":"2022-10-20T14:44:55.269Z","updated_at":"2022-10-20T15:31:38.204Z","monitoring_at":null,"resolved_at":"2022-10-20T15:31:38.190Z","impact":"minor","shortlink":"https://stspg.io/d1wm6mlzdzsj","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"minor","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2022-10-20T14:44:55.264Z","reminder_intervals":null,"id":"0jd10gc3l6bv","page_id":"lhlms7kbdd5n","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2022-10-20T15:31:38.190Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-10-20T15:31:38.190Z","display_at":"2022-10-20T15:31:38.190Z","deliver_notifications":true,"tweet_id":null,"id":"x7d3dk7sdpmr","incident_id":"0jd10gc3l6bv","custom_tweet":null,"affected_components":[{"code":"2n7c98c30hfk","name":"API","old_status":"partial_outage","new_status":"operational"}]},{"status":"identified","body":"We are aware of an issue impacting Teleport Enterprise licensing services. Upon logging into Teleport, users may see a prompt with the following message, “Invalid Teleport license”. Our Engineering team is working quickly to resolve this issue.","created_at":"2022-10-20T14:44:55.335Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-10-20T14:44:55.335Z","display_at":"2022-10-20T14:44:55.335Z","deliver_notifications":true,"tweet_id":null,"id":"6d3qfyr42lq0","incident_id":"0jd10gc3l6bv","custom_tweet":null,"affected_components":[{"code":"2n7c98c30hfk","name":"API","old_status":"operational","new_status":"partial_outage"}]}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"dashboard.gravitational.com Outage","status":"resolved","created_at":"2022-09-14T20:14:43.828Z","updated_at":"2022-09-14T20:36:21.288Z","monitoring_at":"2022-09-14T20:28:52.853Z","resolved_at":"2022-09-14T20:36:21.276Z","impact":"critical","shortlink":"https://stspg.io/38yx6h3t74jl","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2022-09-14T20:14:43.821Z","reminder_intervals":null,"id":"64ztzclnbsm5","page_id":"lhlms7kbdd5n","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2022-09-14T20:36:21.276Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-09-14T20:36:21.276Z","display_at":"2022-09-14T20:36:21.276Z","deliver_notifications":true,"tweet_id":null,"id":"hsqk3kzlcvkn","incident_id":"64ztzclnbsm5","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"operational","new_status":"operational"},{"code":"2n7c98c30hfk","name":"API","old_status":"operational","new_status":"operational"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"operational","new_status":"operational"}]},{"status":"monitoring","body":"A fix has been implemented and we are monitoring the results.","created_at":"2022-09-14T20:28:52.853Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-09-14T20:28:52.853Z","display_at":"2022-09-14T20:28:52.853Z","deliver_notifications":true,"tweet_id":null,"id":"js0w8l693yvt","incident_id":"64ztzclnbsm5","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"major_outage","new_status":"operational"},{"code":"2n7c98c30hfk","name":"API","old_status":"major_outage","new_status":"operational"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"major_outage","new_status":"operational"}]},{"status":"identified","body":"We believe this is related to a database connection issue and are working on a fix.","created_at":"2022-09-14T20:26:15.631Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-09-14T20:26:15.631Z","display_at":"2022-09-14T20:26:15.631Z","deliver_notifications":true,"tweet_id":null,"id":"xy2g1c8hqb4c","incident_id":"64ztzclnbsm5","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"major_outage","new_status":"major_outage"},{"code":"2n7c98c30hfk","name":"API","old_status":"major_outage","new_status":"major_outage"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"major_outage","new_status":"major_outage"}]},{"status":"investigating","body":"We are currently investigating an outage affecting:\n- dashboard.gravitational.com\n- get.gravitational.com\n- dashboard-api.gravitational.com","created_at":"2022-09-14T20:14:43.885Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-09-14T20:14:43.885Z","display_at":"2022-09-14T20:14:43.885Z","deliver_notifications":true,"tweet_id":null,"id":"9w6br55m0025","incident_id":"64ztzclnbsm5","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"major_outage","new_status":"major_outage"},{"code":"2n7c98c30hfk","name":"API","old_status":"major_outage","new_status":"major_outage"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"major_outage","new_status":"major_outage"}]}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Downloads (cdn.teleport.dev)","created_at":"2019-11-25T15:23:48.491Z","updated_at":"2024-07-04T00:33:37.735Z","position":3,"description":"Our file server for downloading releases","showcase":false,"start_date":null,"id":"s0mms0j5qqv9","page_id":"lhlms7kbdd5n","group_id":null}]},{"name":"goteleport.com Unavailable","status":"resolved","created_at":"2022-06-21T07:04:18.440Z","updated_at":"2022-06-21T08:43:13.876Z","monitoring_at":"2022-06-21T07:23:30.255Z","resolved_at":"2022-06-21T08:43:13.862Z","impact":"major","shortlink":"https://stspg.io/23dqnjpx8zn1","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"major","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2022-06-21T07:04:18.435Z","reminder_intervals":null,"id":"66k8g7hxbx1h","page_id":"lhlms7kbdd5n","incident_updates":[{"status":"resolved","body":"Our upstream provider has indicated that this issue is resolved.","created_at":"2022-06-21T08:43:13.862Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-06-21T08:43:13.862Z","display_at":"2022-06-21T08:43:13.862Z","deliver_notifications":true,"tweet_id":null,"id":"fgjhmz1jgtg4","incident_id":"66k8g7hxbx1h","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"degraded_performance","new_status":"operational"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"degraded_performance","new_status":"operational"}]},{"status":"monitoring","body":"We have seen recovery and are monitoring the situation. \n\nWe will provide another update when our upstream provider confirms resolution","created_at":"2022-06-21T07:23:30.255Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-06-21T07:23:30.255Z","display_at":"2022-06-21T07:23:30.255Z","deliver_notifications":true,"tweet_id":null,"id":"wzz0vjxmdk0c","incident_id":"66k8g7hxbx1h","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"degraded_performance","new_status":"degraded_performance"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"degraded_performance","new_status":"degraded_performance"}]},{"status":"identified","body":"We are continuing to work on a fix for this issue.","created_at":"2022-06-21T07:04:31.219Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-06-21T07:04:31.219Z","display_at":"2022-06-21T07:04:31.219Z","deliver_notifications":true,"tweet_id":null,"id":"13yqw5mmjhfw","incident_id":"66k8g7hxbx1h","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"degraded_performance","new_status":"degraded_performance"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"degraded_performance","new_status":"degraded_performance"}]},{"status":"identified","body":"Due to an issue with an upstream provider users my face increased latency, timeouts or unavailability when attempting to navigate https://goteleport.com","created_at":"2022-06-21T07:04:18.522Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-06-21T07:04:18.522Z","display_at":"2022-06-21T07:04:18.522Z","deliver_notifications":true,"tweet_id":null,"id":"ztrgw1vl223q","incident_id":"66k8g7hxbx1h","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"operational","new_status":"degraded_performance"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"operational","new_status":"degraded_performance"}]}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Downloads (cdn.teleport.dev)","created_at":"2019-11-25T15:23:48.491Z","updated_at":"2024-07-04T00:33:37.735Z","position":3,"description":"Our file server for downloading releases","showcase":false,"start_date":null,"id":"s0mms0j5qqv9","page_id":"lhlms7kbdd5n","group_id":null}]},{"name":"Teleport Downloads Throttled","status":"resolved","created_at":"2022-02-07T16:30:48.000Z","updated_at":"2022-05-04T21:44:33.729Z","monitoring_at":null,"resolved_at":"2022-02-08T06:12:14.750Z","impact":"minor","shortlink":"https://stspg.io/58qx3dctf4f3","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2022-02-07T16:30:48.000Z","reminder_intervals":null,"id":"82dg1jflzsjn","page_id":"lhlms7kbdd5n","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2022-02-08T06:12:14.750Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-02-08T06:12:14.750Z","display_at":"2022-02-08T06:12:14.750Z","deliver_notifications":true,"tweet_id":null,"id":"mnk99jmq7zy5","incident_id":"82dg1jflzsjn","custom_tweet":null,"affected_components":[{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"degraded_performance","new_status":"operational"}]},{"status":"investigating","body":"Our initial hypothesis (adding authentication to our s3 proxy) did not improve the download throttling.  We're now investigating other potential mitigations, and continuing to discuss with AWS support In the interim, you may use a direct s3 https download to fetch artifacts:\n\ncurl -O https://s3.amazonaws.com/clientbuilds.gravitational.io/teleport/8.1.4/teleport-8.1.4.pkg\n\nOur next update is anticipated around 2022-02-08 18:00 UTC (10:00 PST).","created_at":"2022-02-08T01:37:37.444Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-02-08T01:37:37.444Z","display_at":"2022-02-08T01:37:37.444Z","deliver_notifications":true,"tweet_id":null,"id":"p4chntl074st","incident_id":"82dg1jflzsjn","custom_tweet":null,"affected_components":[{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"degraded_performance","new_status":"degraded_performance"}]},{"status":"identified","body":"We've identified part of an application that is making unauthenticated API calls to proxy requests to AWS S3. We believe high download volume caused AWS to identify this source of unauthenticated calls as abusive and throttle it.  We're pursuing fixes on two fronts:\n\n1) We've opened an AWS support ticket to temporarily suspend the throttling\n2) We've began researching code changes to the application to have it use authenticated API calls to S3.","created_at":"2022-02-07T21:36:02.007Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-02-07T21:36:02.007Z","display_at":"2022-02-07T21:36:02.007Z","deliver_notifications":true,"tweet_id":null,"id":"5htvqwqs4wrm","incident_id":"82dg1jflzsjn","custom_tweet":null,"affected_components":[{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"degraded_performance","new_status":"degraded_performance"}]},{"status":"investigating","body":"We're currently investigating slow download speeds (50kb/s) from get.gravitational.com.","created_at":"2022-02-07T18:07:48.761Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2022-02-07T21:36:21.767Z","display_at":"2022-02-07T16:30:48.000Z","deliver_notifications":true,"tweet_id":null,"id":"87kgg9kkqxh7","incident_id":"82dg1jflzsjn","custom_tweet":null,"affected_components":[{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"operational","new_status":"degraded_performance"}]}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Downloads (cdn.teleport.dev)","created_at":"2019-11-25T15:23:48.491Z","updated_at":"2024-07-04T00:33:37.735Z","position":3,"description":"Our file server for downloading releases","showcase":false,"start_date":null,"id":"s0mms0j5qqv9","page_id":"lhlms7kbdd5n","group_id":null}]},{"name":"*.gravitational.com certificate expired","status":"resolved","created_at":"2021-12-07T23:00:43.000Z","updated_at":"2022-05-04T21:44:33.799Z","monitoring_at":"2021-12-08T03:02:10.231Z","resolved_at":"2021-12-08T03:18:06.157Z","impact":"major","shortlink":"https://stspg.io/4nwfrnm7qdxm","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2021-12-08T00:00:43.000Z","reminder_intervals":null,"id":"xpk06qz6y1pm","page_id":"lhlms7kbdd5n","incident_updates":[{"status":"resolved","body":"All clear.","created_at":"2021-12-08T03:18:06.157Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2021-12-08T03:18:06.157Z","display_at":"2021-12-08T03:18:06.157Z","deliver_notifications":true,"tweet_id":null,"id":"h4cdmq416yj4","incident_id":"xpk06qz6y1pm","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"operational","new_status":"operational"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"operational","new_status":"operational"}]},{"status":"monitoring","body":"New certs deployed.  dashboard.gravitational.com and get.gravitational.com look to be working, but we're monitoring for the next 15 minutes.","created_at":"2021-12-08T03:02:10.231Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2021-12-08T03:02:10.231Z","display_at":"2021-12-08T03:02:10.231Z","deliver_notifications":true,"tweet_id":null,"id":"qhrzjnxszmcd","incident_id":"xpk06qz6y1pm","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"partial_outage","new_status":"operational"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"partial_outage","new_status":"operational"}]},{"status":"identified","body":"We've provisioned the new certificate and are deploying it.","created_at":"2021-12-08T01:49:34.191Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2021-12-08T01:49:34.191Z","display_at":"2021-12-08T01:49:34.191Z","deliver_notifications":true,"tweet_id":null,"id":"bjmx6ld0brqm","incident_id":"xpk06qz6y1pm","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"partial_outage","new_status":"partial_outage"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"partial_outage","new_status":"partial_outage"}]},{"status":"investigating","body":"Team alerted, we're provisioning and deploying a new certificate.","created_at":"2021-12-08T01:10:50.616Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2021-12-12T20:11:18.199Z","display_at":"2021-12-08T01:10:50.000Z","deliver_notifications":true,"tweet_id":null,"id":"grjzy0679tbt","incident_id":"xpk06qz6y1pm","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"operational","new_status":"partial_outage"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"operational","new_status":"partial_outage"}]},{"status":"investigating","body":"*.gravitional.com certificate expired. dashboard.gravitational.com and get.gravitational.com (backing teleport downloads) begin throwing certificate errors.","created_at":"2021-12-08T01:09:43.848Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2021-12-12T20:09:38.150Z","display_at":"2021-12-08T00:00:43.000Z","deliver_notifications":true,"tweet_id":null,"id":"ldpv6h36grg6","incident_id":"xpk06qz6y1pm","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"operational","new_status":"operational"},{"code":"s0mms0j5qqv9","name":"Downloads","old_status":"operational","new_status":"operational"}]}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Downloads (cdn.teleport.dev)","created_at":"2019-11-25T15:23:48.491Z","updated_at":"2024-07-04T00:33:37.735Z","position":3,"description":"Our file server for downloading releases","showcase":false,"start_date":null,"id":"s0mms0j5qqv9","page_id":"lhlms7kbdd5n","group_id":null}]},{"name":"System Outage","status":"resolved","created_at":"2019-11-27T19:43:43.824Z","updated_at":"2022-05-04T21:44:33.865Z","monitoring_at":null,"resolved_at":"2019-11-27T19:57:43.824Z","impact":"critical","shortlink":"https://stspg.io/b0cfr0s4b0xs","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"critical","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2019-11-27T19:43:43.824Z","reminder_intervals":null,"id":"dpylwgp9fscg","page_id":"lhlms7kbdd5n","incident_updates":[{"status":"resolved","body":"The system was unavailable after a wrong commit was deployed and had to be rolled back.","created_at":"2019-11-27T20:44:43.824Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2019-11-27T21:15:46.805Z","display_at":"2019-11-27T19:57:43.824Z","deliver_notifications":true,"tweet_id":null,"id":"k73l03mwwrlh","incident_id":"dpylwgp9fscg","custom_tweet":null,"affected_components":[{"code":"j02c9gqk79zm","name":"Dashboard","old_status":"operational","new_status":"operational"},{"code":"2n7c98c30hfk","name":"API","old_status":"operational","new_status":"operational"},{"code":"s0mms0j5qqv9","name":"File Server","old_status":"operational","new_status":"operational"}]}],"postmortem_body":"## Summary\n\nAll customer-facing components were affected by an outage that lasted approximately 20 minutes. The root cause was a bad commit that got deployed by an automated system and had to be rolled back. Our monitoring systems detected the outage and notified the responsible team, which then proceeded to initiate a rollback.\n\n## Root cause\n\nWhile testing an upcoming release, our build system incorrectly tagged an artifact as production ready. This was not intended to be deployed, however an unrelated system crash and subsequent restart in our production environment lead to this artifact being pulled and used.\n\n## Diagnosis\n\nAfter our monitoring systems notified the responsible team, we could quickly confirm our intuition by consulting the system and event logs. The rollback process was initiated and carried out by our build servers through an automated process.\n\n## Timeline\n\n19:37 - Kubernetes killed a running container because of memory constraints\n\n19:38 - The newly created container was stuck in a crash-loop as it was expecting a new configuration flag to be present\n\n19:43 - The team was notified as too many subsequent health-checks failed\n\n19:52 - The root cause was identified\n\n19:53 - The rollback process was initiated\n\n19:57 - All systems were operational again\n\n## Prevention\n\nWe identified this failure scenario a few days prior as part of a routine review of our deployment process. This issue was addressed, but not yet merged at the time of this outage. \n\nFurther, we will start collecting additional metrics to get a better understanding what lead to the memory constraints that ultimately triggered this failure.","postmortem_body_last_updated_at":"2019-11-27T21:12:41.124Z","postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Downloads (cdn.teleport.dev)","created_at":"2019-11-25T15:23:48.491Z","updated_at":"2024-07-04T00:33:37.735Z","position":3,"description":"Our file server for downloading releases","showcase":false,"start_date":null,"id":"s0mms0j5qqv9","page_id":"lhlms7kbdd5n","group_id":null}]}]
}