tag:status.stord.com,2005:/historyStord Status - Incident History2024-03-29T01:22:53-04:00Stordtag:status.stord.com,2005:Incident/203527652024-03-25T12:28:00-04:002024-03-25T13:18:06-04:00WMS API Certificate Renewal<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>12:28</var> EDT</small><br><strong>Resolved</strong> - WMS API https://api.wms.stord.com TLS/SSL certificate automated renewal request was unsuccessful without notification from certificate service. We were able to recover quickly by processing and applying the new certificate. We will implement robust proactive monitoring of our services' TLS/SSL certificate expirations to prevent this from happening again. We are committed to continuous improvement and are dedicated to providing our customers with the most reliable service possible.</p>tag:status.stord.com,2005:Incident/202376042024-03-24T06:00:56-04:002024-03-24T06:00:56-04:00Stord OMS Database Maintenance<p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>06:00</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>05:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>17:14</var> EDT</small><br><strong>Scheduled</strong> - The database will be undergoing mandatory maintenance upgrades during this time. Total downtime is estimated to be less than five minutes once the maintenance begins.</p>tag:status.stord.com,2005:Incident/200379752024-02-21T08:50:38-05:002024-02-21T08:50:38-05:00WMS is not currently accessible<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>08:50</var> EST</small><br><strong>Resolved</strong> - We have identified and resolved the issue. All users should be able to successfully log-in to Stord WMS and access functions and pages within the WMS.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>08:33</var> EST</small><br><strong>Identified</strong> - We believe we have identified the issue and are working on a fix.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>07:37</var> EST</small><br><strong>Investigating</strong> - We are aware that users are unable to log in to Stord WMS are are currently investigating this issue.</p>tag:status.stord.com,2005:Incident/191986752023-11-23T05:01:24-05:002023-11-23T05:01:24-05:00Scheduled Maintenance<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>05:01</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>04:45</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>19:21</var> EST</small><br><strong>Scheduled</strong> - During this window, there will be temporary, planned downtime. Data flows will resume from the point of maintenance immediately following without any decline in service.<br /><br />This maintenance is proactive in nature for the continued scale and growth of the Stord platform. The maintenance window will be used to implement improvements to our infrastructure to continue providing a highly reliable platform.<br /><br />We will provide updates on https://status.stord.com If you have any questions or concerns, please contact us through https://help.stord.com.<br /><br />Thank you for your understanding.<br />The Stord Team</p>tag:status.stord.com,2005:Incident/189716102023-10-30T16:41:23-04:002023-10-30T16:41:23-04:00Stord Domain Outage<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>16:41</var> EDT</small><br><strong>Resolved</strong> - Our provider has resolved the issue</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>16:24</var> EDT</small><br><strong>Monitoring</strong> - Service has been restored. A third party provider had a temporary outage causing our sites to be unreachable. We are continuing to monitor the status of this provider.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>16:14</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>16:10</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>16:10</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>16:09</var> EDT</small><br><strong>Investigating</strong> - We are continuing to investigate an issue affecting all stord.com sites. Our APIs remain operating normally, and orders continuing to process. An update will be provided no more than 30 minutes from now.</p>tag:status.stord.com,2005:Incident/186435722023-10-15T05:46:56-04:002023-10-15T05:46:56-04:00Scheduled Maintenance<p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>05:46</var> EDT</small><br><strong>Completed</strong> - Scheduled outage is complete. All systems are up and available at this time. The approximate period of downtime took place between 9:04 am and 9:17 am UTC.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>05:01</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>11:27</var> EDT</small><br><strong>Scheduled</strong> - Stord OMS and Stord WMS are scheduled for routine maintenance on Sunday, October 15th, 2023 from 9:00 am to 10:00 am UTC.<br /><br />During this window, there will be temporary, planned downtime. Data flows will resume from the point of maintenance immediately following without any decline in service.<br /><br />This maintenance is proactive in nature for the continued scale and growth of the Stord platform. The maintenance window will be used to implement improvements to our infrastructure to continue providing a highly reliable platform.<br /><br />We will provide updates on https://status.stord.com If you have any questions or concerns, please contact us through https://help.stord.com.<br /><br />Thank you for your understanding.<br />The Stord Team</p>tag:status.stord.com,2005:Incident/186666622023-10-03T15:48:29-04:002023-10-03T15:48:29-04:00Delayed Data Freshness<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>15:48</var> EDT</small><br><strong>Resolved</strong> - Root cause has been resolved. Data freshness will be continually monitored and additional optimizations made as needed over the coming days.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>11:15</var> EDT</small><br><strong>Update</strong> - We are continuing to experience intermittent delays in order data freshness. Further efforts are being taken to ensure data remains fresh during periods of higher throughput. These efforts are expected to complete before end of day. A status update will be provided by 4PM ET today.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>09:50</var> EDT</small><br><strong>Monitoring</strong> - An incident was resolved 10/1, resulting in delayed order data to our OMS dashboards, order search pages, and public APIs for order retrieval. We expect delays in order data freshness until 10/2 at 9pm ET.</p>tag:status.stord.com,2005:Incident/168916872023-04-30T07:00:04-04:002023-04-30T07:00:04-04:00Scheduled maintenance<p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>07:00</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>06:12</var> EDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>05:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>11:02</var> EDT</small><br><strong>Scheduled</strong> - Stord OMS and Stord WMS are scheduled for routine maintenance on Sunday, April 30th, 2023 from 9:00 am to 11:00 am UTC. <br /><br />During this window, there will be temporary, planned downtime. Data flows will resume from the point of maintenance immediately following without any decline in service.<br /><br />This maintenance is proactive in nature for the continued scale and growth of the Stord platform. The maintenance window will be used to implement improvements to our infrastructure to continue to provide a highly reliable platform.<br /><br />We will provide updates on status.stord.com. If you have any questions or concerns, please contact us through help.stord.com.<br /><br />Thank you for your understanding.<br />The Stord Team</p>tag:status.stord.com,2005:Incident/163919072023-03-08T14:10:00-05:002023-03-08T14:10:00-05:00Elevated error rates on the Connect platform for some customers<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>14:10</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved and all systems have been stable since 1:39 pm ET.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>12:37</var> EST</small><br><strong>Monitoring</strong> - An issue for some Integration connections for some customers was identified between 12:37 pm ET and 1:39 pm ET.<br />A fix has been implemented and our engineers are monitoring for continued stability.</p>tag:status.stord.com,2005:Incident/153842192022-12-17T00:56:37-05:002022-12-17T00:56:37-05:00Event Processing Degraded Performance<p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>00:56</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>00:55</var> EST</small><br><strong>Update</strong> - The cloud infrastructure provider has indicated the incident has been resolved, and Stord has confirmed connectivity appears to have returned to normal. Stord will be investigating impact and addressing any errors.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>00:40</var> EST</small><br><strong>Update</strong> - The cloud infrastructure provider has implemented its fix and is monitoring the results. Stord is still observing some instability while the provider's infrastructure stabilizes but will continue to monitor.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>20:21</var> EST</small><br><strong>Monitoring</strong> - One of Stord's cloud infrastructure providers is encountering issues which may lead to intermittent delays for all customers in processing of messages between systems, including creation and fulfillment of orders. Fixes are underway by the provider and additional status updates will be provided as they become available.</p>tag:status.stord.com,2005:Incident/128735432022-11-16T08:06:09-05:002022-11-16T08:06:09-05:00Stord to Expand IPs for Egress Connection<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>08:06</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>08:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>17:18</var> EST</small><br><strong>Scheduled</strong> - Stord Engineering will perform a one time change to ensure the platform remains reliable during elevated traffic levels. <br /><br />Our current IP address range for egress connections is a single IP: <br />- 35.231.249.93<br /><br />On Wednesday November 16th, 2022 at 8 am ET, Stord will broaden our IP address range to: <br />- 35.231.249.93<br />- 34.148.114.49<br />- 34.139.221.177<br />- 35.190.190.29<br /><br />There is no downtime for this change. <br /><br />From our internal testing and monitoring we do not expect any interruption to service.<br /><br />Though unlikely, please review your network restrictions in the context of the IP address expansion, and update them as necessary by November 16th, 8 am ET. <br /><br />Please Let us know if you have any questions or concerns at support@stord.com and we will be happy to address them.</p>tag:status.stord.com,2005:Incident/126239742022-10-30T12:20:17-04:002022-10-30T12:20:17-04:00Stord One Commerce Maintenance<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>12:20</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>12:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>09:58</var> EDT</small><br><strong>Scheduled</strong> - This Sunday (10/30/2022) at noon EDT there will be scheduled downtime for approximately 30 minutes. The Stord One Commerce API and Application will be briefly unavailable. <br />All operations around the platform will resume immediately after the window.</p>tag:status.stord.com,2005:Incident/126242202022-10-26T14:18:08-04:002022-10-26T14:18:08-04:00Network-wide issues with UPS<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>14:18</var> EDT</small><br><strong>Resolved</strong> - UPS has triaged and resolved their outstanding issues. <br />The Stord UPS account is printing labels as expected and all buildings are operational.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>10:23</var> EDT</small><br><strong>Investigating</strong> - We are aware of ongoing, network-wide issues with UPS; please expect a delay in order processing and shipping for orders shipping via UPS. We will provide updates as we work with UPS.<br /><br />Please reach out to your Customer Success Manager with any questions.</p>tag:status.stord.com,2005:Incident/113026612022-09-28T19:25:50-04:002022-09-28T19:25:50-04:00Delay for some customers due to vendor outage<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>19:25</var> EDT</small><br><strong>Resolved</strong> - The Stord team has been monitoring the external services for the last two hours and appears to be running normally. The team has already rectified any major inconsistencies and will continue to monitor. If there is further action required, customers may be contacted directly.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>17:09</var> EDT</small><br><strong>Update</strong> - The AWS team has given the all-clear for the incident as of 2:05pm Pacific, and our integration vendor has indicated that services are back running as normal. The Stord team will be executing its outage recovery playbook to ensure that all affected orders and fulfillments are properly processed; however, the integration vendor has indicated it may be 24-48 hours before the backlog may be processed in its entirety. An update will be provided once the recovery protocol has been completed.<br /><br />Earlier today it was reported that Amazon Web Services was experiencing degraded performance in services in its us-west-2 region. One of Stord's integration vendors utilizes AWS, so Stord's connection to that vendor had been exhibiting degraded performance as well. As a result of this, some of Stord's customers may have experienced delays in being able to view new orders in Stord One Commerce and seeing fulfillments posted back to their storefront.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>16:45</var> EDT</small><br><strong>Update</strong> - The Stord team continues to actively monitor the issue, and the latest update from AWS at 1:22PM Pacific time indicates a reduction in errors due to additional mitigations put in place, with more mitigations being applied over the next half hour. The Stord team has seen a brief improvement in performance, but has not yet fully recovered.<br /><br />Earlier today it was reported that Amazon Web Services was experiencing degraded performance in services in its us-west-2 region. One of Stord's integration vendors utilizes AWS, so Stord's connection to that vendor has been exhibiting degraded performance as well. As a result of this, some of Stord's customers may be experiencing delays in being able to view new orders in Stord One Commerce and seeing fulfillments posted back to their storefront.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>15:36</var> EDT</small><br><strong>Update</strong> - The Stord team continues to actively monitor the issue, and the latest update from AWS at 12:26PM Pacific time indicates identification of the root cause and continued on-going work to resolve, with hopeful resolution within two hours (2:30PM Pacific).<br /><br />Earlier today it was reported that Amazon Web Services was experiencing degraded performance in services in its us-west-2 region. One of Stord's integration vendors utilizes AWS, so Stord's connection to that vendor has been exhibiting degraded performance as well. As a result of this, some of Stord's customers may be experiencing delays in being able to view new orders in Stord One Commerce and seeing fulfillments posted back to their storefront.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>14:58</var> EDT</small><br><strong>Monitoring</strong> - Earlier today it was reported that Amazon Web Services was experiencing degraded performance in services in its us-west-2 region. One of Stord's integration vendors utilizes AWS, so Stord's connection to that vendor has been exhibiting degraded performance as well. As a result of this, some of Stord's customers may be experiencing delays in being able to view new orders in Stord One Commerce and seeing fulfillments posted back to their storefront. The Stord team continues to actively monitor the issue, and the latest update from AWS at 11:33AM Pacific time indicates identification of the root cause and continued on-going work to resolve.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>14:11</var> EDT</small><br><strong>Update</strong> - Some service calls to the external vendor are starting to succeed. The Stord team is continuing to monitor.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>13:50</var> EDT</small><br><strong>Identified</strong> - One of Stord's integration vendors has identified an outage in their Amazon Web Services components. As a result of this, customers that are integrated through this vendor may observe delays in orders and fulfillment notifications.<br /><br />No workarounds are available at this time. The Stord team is actively monitoring the situation and will be ready to assess any further impact once the vendor's services have been fully restored. This page will be updated once more information is available, but as of 2:10pm Eastern AWS is currently not able to provide an ETA as of their 10:59am Pacific update.</p>tag:status.stord.com,2005:Incident/108327252022-08-15T17:00:38-04:002022-08-15T17:00:38-04:00Degraded performance of 3rd Party Data Streaming Infrastructure<p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>17:00</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>16:55</var> EDT</small><br><strong>Monitoring</strong> - Connections appear to be stable again and we are continuing to monitor.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>16:21</var> EDT</small><br><strong>Identified</strong> - Conditions<br />- Stord One Commerce and WMS are experiencing intermittent disconnects to our 3rd party data streaming infrastructure, Confluent Cloud. <br />- This can impact orders, inbounds, and outbounds across both platforms.<br />Actions<br />- Our Engineers are monitoring the situation and reconnecting as soon as possible. Confluent Cloud has not acknowledged any public incident.<br />Workaround<br />- No workaround at this time.<br />Next Update<br />- At one hour unless conditions change.</p>tag:status.stord.com,2005:Incident/85524022021-11-16T13:14:58-05:002021-11-16T13:14:58-05:00Our hosting provider Google Cloud Platform, is experiencing severe network issues.<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>13:14</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>13:13</var> EST</small><br><strong>Monitoring</strong> - Incident appears to be resolved. More information about the issue can be found here.<br />https://twitter.com/GCP_Incidents/status/1460671761431859202</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>13:12</var> EST</small><br><strong>Identified</strong> - Verified that incident is due to networking issues with our underlying cloud provider.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>13:04</var> EST</small><br><strong>Investigating</strong> - We are noticing that Spotify and other vendors using Google Cloud Platform are experiencing outages.</p>tag:status.stord.com,2005:Incident/78025602021-08-19T19:08:58-04:002021-08-19T19:08:58-04:00Degraded Performance from Authentication and Authorization Provider<p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>19:08</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>18:14</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing degraded performance from our Authentication and Authorization provider. This is affecting the ability for users to log in. This is currently being investigated.</p>tag:status.stord.com,2005:Incident/73143262021-06-23T12:21:30-04:002021-06-23T12:21:30-04:00Login to the application is blocked.<p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>12:21</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>12:20</var> EDT</small><br><strong>Identified</strong> - We have identified the issue and are resolving it now.</p><p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>12:18</var> EDT</small><br><strong>Investigating</strong> - We are addressing this right now and it will be fixed in 5 minutes or less.</p>tag:status.stord.com,2005:Incident/67978272021-04-20T18:26:16-04:002021-04-20T18:26:16-04:00Authenticating to the application is not working<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>18:26</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>16:35</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>14:08</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>14:08</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>14:07</var> EDT</small><br><strong>Investigating</strong> - Auth0.com is having a major outage that is preventing logins from working. The system is processing transactions normally, only access to the user interface is impacted.</p>tag:status.stord.com,2005:Incident/59773012021-01-08T10:03:41-05:002021-01-08T10:03:41-05:00Customers unable to login<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>10:03</var> EST</small><br><strong>Resolved</strong> - The issue has been resolved.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>10:02</var> EST</small><br><strong>Identified</strong> - The issue has been identified and we are taking steps to resolve it immediately.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>10:01</var> EST</small><br><strong>Investigating</strong> - Customers are unable to login to the application due to permission issues.</p>tag:status.stord.com,2005:Incident/59772662021-01-08T09:53:50-05:002021-01-08T09:59:13-05:00Customers unable to login<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>09:53</var> EST</small><br><strong>Resolved</strong> - Customers were not able to login after a recent release. A rollback of the release resolved the issue. We will perform a respective to ensure this type of issue does not occur again.</p>tag:status.stord.com,2005:Incident/48677122020-08-15T14:33:20-04:002020-09-15T14:00:11-04:00This is an example incident<p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>14:33</var> EDT</small><br><strong>Resolved</strong> - Empathize with those affected and let them know everything is operating as normal.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>14:23</var> EDT</small><br><strong>Monitoring</strong> - Let your users know once a fix is in place, and keep communication clear and precise.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>14:29</var> EDT</small><br><strong>Identified</strong> - As you continue to work through the incident, update your customers frequently.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>14:59</var> EDT</small><br><strong>Investigating</strong> - When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.</p>