Outages and Domino Art
There they are in front of you. Ten thousand dominos standing on the floor… waiting. You have the perfect viewing location in the front row of the balcony overlooking the event. Then it happens. Someone pushes the first domino. Artistically and with predictable rhythm the dominos begin to fall. Each one striking the next one as if to say “It’s your turn.” Then within five minutes, it is all over. The last domino falls as if taking a bow to the cheering of the crowd.
Domino art is a one-time unique event that depends entirely on the skill of the setter. If only one domino is out of place, too close or too far away from those on either side, the art fails. Each domino is dependent on two others to fulfill the design of the artist. The beauty of domino art is only fully appreciated as all of the dependencies are present and doing their job.
“Blink is down again! You gotta be kiddin’ me. What kind of software are we running here? It seems like it is down more than it is running!” Perhaps you have heard that statement in the last couple of weeks or even said it yourself.
And as we’ve run through a rough patch keeping all systems go, it made me stop and marvel at the many dependencies that need to be in place to keep Blink up and running, producing its artistry, on a 24/7 schedule.
Recently while working on the functional code for Blink, I copied a command from our documentation and ran it in our production environment without closely examining the command or its parameter list. After all, it was from our documentation! The result was a loss of all Blink icons for about four hours. Programming without error? In heaven maybe! Common sense programming is something Blink relies on to keep chugging each and everyday (and using common sense is as close to error free as we can get in this life).
Authentication (what you use every day or every hour to log in to Bethel’s systems) is the second dependency needed to keep Blink running. All this work to log community members in and out on a continual basis is done by two authentication servers – machines that do nothing but authenticate users and send a validation token to Blink that says “this person is a legitimate user… let them in.” If one of the authentication servers has a problem, a critical dependency is lost and the dominos will be stopped cold.
Blink also depends on data flow. To serve you information that’s geared to your role at Bethel, Blink constantly draws info from multiple databases. Without that data flow, Blink will display the familiar “Error Report” instead of the channel information that you expect to see or worse, Blink will shut down entirely. Each bit of data in turn depends on database operating systems and network infrastructure to transmit a proper sequence of info and at just the right time.
Blink rarely shuts down just because it is tired or grumpy or having a bad hair day. Like domino art, Blink relies on many other dominos doing exactly what they were intended to do when they were intended to do it. When that happens consistently, the artistry of our portal will continue to run 24/7 without interruption. It’s something I sometimes take for granted as web systems become more and more a part of our everyday at Bethel and beyond–all dependencies, working together, performing their function in the proper order. It really is a thing of beauty, something to be cheered.