The Email Said 'People Will Die'

The management at my company received an email on Thursday that stated people will die if we continue with our planned software update this weekend.

I have been keeping a personal diary about my thoughts and the daily news headlines I see on the internet regarding the Coronavirus. A few years from now it will be interesting to read and I think my grand kids will enjoy reading my personal thoughts about something they are likely to be taught in school.

But that is for another day.

I work for a company you will not recognize, and I don't want to mention yet. We are a SaaS company. Software as a Service. Kind of like Google, Amazon or Facebook. No software is downloaded to your PC, phone or tablet by a SaaS company. You have a username and password and you log in to use their services.

I am Senior Full Stack Java Developer.

We provide many boring business services so I will keep it simple. Our customers include most of the major universities and many governments and large companies. We are an international company and you would recognize many of our customers. One of our features is eProcurement, kind of like Amazon. For example, in a large university you have professors, researchers, administrators, etc all purchasing goods and services. Those could be printer paper, computers, catering, mice for research or even radioactive materials. The university will have negotiated contracts with vendors for these items.

They might have a contract with Staples for computer related equipment at a volume discount. People in the university will use our service to shop for computers from Staples, create a shopping cart in our service and then create a Purchase Requisition (PR). This PR will be routed electronically for approvals in the university administration based on amounts, account codes or even the person creating the PR. When finally approved the PR creates a Purchase Order (PO) which is sent electronically to Staples to be fulfilled.

We do many other things but I will explain one more thing because it relates to this story. Some businesses, like hospitals, have supply rooms. A hospital might have many items like gloves, masks, needles or drugs in a supply room near the receiving dock at the building. Those items are needed in many areas of the hospital so they need a system to requisition items from the supply room to specific locations in the building. We have a feature for that. A user logs into our system to request the items they need and a hospital employee delivers them. Our system can also automatically order more of an item when the inventory gets low. There is no paperwork or phone call. It is all online.

A SaaS company is always adding new features to their service. The two largest components that are modified for new features are the database and the software that reads and writes to that database. That usually requires the service to be taken off line for some period of time as the database updates can take several minutes to hours. The software may have to be deployed to many web server computers. If the service is running during these updates the users will get errors and data corruption can occur.

Our customers pay subscription fees and expect our service to work correctly all the time or their business could suffer.

So three times a year like clockwork we update our service with new features. The service is taken offline at 9:00 PM on a Friday and updated. The development teams start testing at 8:00 AM on Saturday to validate that all is good and usually the service is back online sometime late Saturday. I think our contractual obligation is to have the service back online by noon on Sunday.

In my 11 years with this employer the process has never changed. The customers know about these outages and plan for them as they are published a year in advance and the dates never change. They prepare and train their users for the new features.

On Thursday our management received a email from one of our customers, a large hospital in the American north east area. They stated that if we went ahead with our planned outage and update this weekend that PEOPLE WILL DIE. Four field hospitals are being set up to treat Coronavirus victims and they use our supply room feature as well as the PR/PO stuff I explained earlier to order urgently needed supplies among many other features.

They want the update postponed until a later date. We have never done that in the history of the company. There are new features in this release that other customers need. Also there will be no better weekend to do this update in the future. Next weekend there will be some other customer wanting to postpone the update.

Our management came up with a plan. We will go offline Friday night at 9:00 PM as planned. The software and database updates usually take about three hours. My team and another will do some quick sanity testing at midnight and the service will be placed back online. Hopefully the downtime will be 3-4 hours instead of 24 hours or more. The other development teams will do their testing at 5:00 AM on Saturday instead of 8:00 AM, after the updates are ALREADY LIVE.

If someone had proposed doing this one month ago they would have been laughed out of the office. Software is complex and we always find a few issues during the Saturday morning testing that must be fixed before placing the service back online.

Next week will start the 5th week of working from home. I have not left our 2 acres in over a week. There is so much doom and gloom on the news but today was an absolutely beautiful day here in NC with deep blue skies, a cool breeze and budding trees.

This is so surreal. The clock in the photo at the top of this blog helps me to remember the day of the week since the days tend to run together.

Stay safe and wash your hands!