Monday, July 24, 2023

Hot Take: IT is the lifeblood of companies.

I have wanted to write about this for some time, it's an idea that, when you really think about it, it is both true and almost scary.  Simply put, modern business does not function without I.T.; as IT, I am the lifeblood of the company.  Horrifically, most business owners don't understand this, so I will explain.

As a preamble, this is a long, rant-y post, and it's off-the-cuff, so please take it as it is, and forgive my (I'm sure, plenty of) grammar and spelling transgressions.

In the past, before computerization, companies largely relied on paper; forms filled out by various individuals along the chain, adding to each document as things went along, creating a "paper trail" of activities, overseen by managers and decision-makers to make things go.  The sales cycle would go from telephones and in-person discussions with clients, to a quote, to an order, to fulfillment, which may or may not go through manufacturing steps, each with their own related paperwork, then to final delivery, which then gets passed over to accounts receivable for final collection.  At the end of the day, the money collected goes over to finance, who pays everyone for their work, and pays the bills to companies that provided materials, all of which have this same process happening internally.  The cycle continues, more sales are made, and more products are delivered.  This is the business process.

Presently, every step of this, aside from perhaps manufacturing, has been digitized, and relies on IT to work.  Sales are done by a combination of phones (often over VoIP), E-Mail, and in-person discussion (usually coordinated by the first two), the quote is prepared digitally on some sales software, sent electronically, and often approved/signed digitally.  That invoice is now sent, digitally, over to the fulfillment, which if there's manufacturing involved, every step of manufacturing is often tracked and sometimes even performed by or on computers.  Final delivery is generally arranged online for physical deliveries, whether through your parcel service of choice, or to schedule in-house drivers and get the product to the client; additionally for parcels, if you're not scheduling the pickup online, you're probably doing it by phone, which is VoIP, which is ALSO I.T. DRIVEN; then the final collection is usually done with a combination of email and phones, payment is shifting away from physical cheques to EFT, so now that process is entirely on the computer, and accounting/payroll is done entirely on the computer as well.

This is how most companies are going... "paperless"; even product resellers are using this process, in addition to inventory tracking, etc, to get things going, to the point at which there are WiFi connected devices that are scanning and tracking the inventory and ordering more.  The entire process, top to bottom, at every step, has I.T. involvement, and it's not slowing down; steps that are not computer driven are becoming computer driven.  Management has instantaneous access to workers punch-in and punch-out data, knowing who is on-site and working at all times, they're easily able to look up the status of any order and where it is in the chain of delivery to the client, from the deliverable being manufactured on the shop floor, to where the delivery driver is physically located right this second.  Management has an unparalleled amount of transparency into what workers are currently doing, how things are moving along, and when delivery to the client's door will happen.

This entirely relies on I.T. 100%.  Companies have slowly transformed from their specific market, a company selling x product, into data management organizations, that make money by selling x.  Everything from CRM, marketing, sales, accounting/finance, management, communication in all forms, etc, all driven by the network, and servers.

Make no mistake, any modern business IS A DATA MANAGEMENT COMPANY; they just happen to make money by selling a specific product or set of products, either through manufacturing it, or reselling it.

Every step of that process is I.T. driven.  Networking is the core of the information systems.  As a networker, I know this intimately. As an associate of mine (also a networker) has said previously:

"Networking is the bottom of the stack; no one cares about you, but everyone depends on you.  You will be the first point anyone looks to when there is a problem and be expected to provide input and advice about everything running on the network whether that's your job or not.  You need to know maths (subnets/latency), be able to visualize data flows in your head, and be able to build solutions with sub-second resiliency.  If you are ready for that, then come aboard. I'll pour the scotch."  - Laz Pereira

Laz is absolutely correct here; he's an excellent networker, and someone that a company could only be so lucky to land as a technical resource.  The fact is, when I.T. started, it was a convenience that mutated into a core competency for any thriving modern business.  Every sector, seemingly without exception, is computerized, relying on servers which rely on networking.  The problem is, since it was historically seen as a convenience, many (especially older) business execs still see it that way, as a convenience, not a requirement.  Meanwhile, without I.T. productivity would grind to a halt.  Imagine if Amazon, the largest product reseller out there, at least for their front-end business, suddenly had a major I.T. outage at a distribution center.  The workers, who are massively over-worked and my heart goes out to them, have their entire life dictated by the whims of their I.T. system.  They're expected to collect all the orders in a timely fashion, and are timed for their collection times based on where they are, and how long the system thinks it should take to get to the next item for collection.  This extends to their drivers, who are equally pushed to complete a number of items per hour delivered, all governed by a device they carry.  Whether you're inside the warehouse or in a truck, a small device connected to a large system of servers via a network infrastructure, dictates everything they do.  If that system, or the network on which it resides, was to be interrupted for even a few hours, their business would GRIND TO A HALT during that time.

Amazon is an extreme example, but simply, they have the infrastructure to ensure that instances of down-time are minimized into irrelevance, but other companies have the same problem, and do not have the same protections of redundancy.  There are huge swaths of industries that are ill-equipped for a major IT outage.  I'll give you a real-world example; I heard this third-party, but I have high confidence that the important details are correct.

A local business-owner, whose business manufactured plants, of all things - mainly growing them from seeds and distributing either fully-grown plants or clippings for sale, had a major loss and significant down-time.  What happened taught a powerful lesson to the owner.  The company suffered loss by fire.  Their primary office complex caught fire.  I want to be clear, they own acres of land and have several manufacturing, processing and distribution buildings, they run their own delivery service for their products, and they're very vertically integrated.  As with most businesses, they had tracking for manufacturing, using WiFi and small devices for inventory tracking (like hand-held scanners), VoIP, and their entire office structure was based on a paper-less system, so 90% of internal communication for orders, sales, invoicing, accounting, etc. was all done entirely digitally.  During the fire, as I was told, they had IT support remoted into servers, WHILE THE BUILDING WAS ON FIRE, trying to back up all the data they could.  Unfortunately, up until that point, the company owner viewed IT as more of a cost center, than a critical business asset, like it is.  The key point of this story is that when they rebuilt the building and got back to business-as-usual, as part of the rebuild, IT was now a key focus.  I worked with the business after the fact to help with redundancy in their IT infrastructure.

The lessons learned by the company owner drove him to have on-premise (different building) network-attached backups, as well as off-site sync for those backups, and a backup compute cluster for his primary servers.  He got about 80% of the way to full redundancy; literally able to shut down the main servers, fire up the backups, which were pre-emptively restored to the backup cluster every hour, and be up and running with little more than one hour of lost data.  The issue I saw, as a networker, was that their network was severely vulnerable.  They had dual internet connections, on a high-availability dual firewall set-up, which were geographically located in different buildings.  The issue in all of this was that all of their inter-building connections were run through a single structure; they were using largely a single VLAN for most of their connectivity, and since all of the ISP connections they were using, while redundant, were in out-buildings, if the primary building was lost, ALL connectivity to the ISPs and other buildings would be lost.  The data would be intact, and the servers would operate, but with no internet connection, and no connection to any other building in their campus.  I tried to warn them of this and propose solutions, but the discussion fell flat and I never got to the point where I could convince them to invest in the new data connectivity required to bring the system to be fully redundant; another technician took over, who was not a networker, and I don't think he understood that a problem existed, and as far as I know, the system is still vulnerable to this; which is why I won't mention the company name.  I don't want to call them out for making this error, at the end of the day, the company owner is still responsible for this oversight, and for hiring someone who doesn't have the ability to visualize the data flow, and understand where the data is moving to/from which would make the problem very obvious.

Fact is, I.T. is still seen as a cost center, and it's not.  It's become a critical business asset which needs to be taken as seriously as worker productivity.  Companies like Amazon understand this, but it's unsurprising that a multi-national monster like Amazon would.  They built-up AWS, in no small part, to support themselves; until AWS came around, there were not many cloud-based systems with the breadth and depth to support Amazon's growth.  Not all companies understand how critical it can be to have highly available, reliable and fast access to their data.  The contrast of company owners wanting to save money on the cost-center that is I.T. and what I.T. is actually doing for them, and how critical I.T. is for modern business, is staggering.

From personal experience, as some of you may know, I work IT support right now, working mainly at an MSP, which basically inserts me into companies as their de-facto I.T. guy.  It's hard for me to express the frustration of trying to do the best thing for my customer, and getting push-back on it from every side.  Whether it's on a new network switch which fits the customer needs but the sales team thinks is "too expensive" and the client would never approve it, to server infrastructure and systems, that my coworkers think is "unsupportable" because they don't understand it, and are not willing to learn it.  I'm constantly pushing for better and better equipment, knowing that a companies trajectory can turn on a dime; explosive growth could obsolete the capabilities of a system within a few years, well within the warranty period.  Planning for that potential growth and over-shooting what actually happens is rarely a bad thing; additionally, investing $1000, or even $10000 per switch in a data-critical environment is not even expensive for most businesses, it's just more than they would like, often opting for inexpensive solutions that barely meet their needs today, or are worse than their current equipment.  I've witnessed the removal of several high-end systems to be replaced with newer, but worse, solutions all because sales and company leadership doesn't understand the impact they're having on the infrastructure that underpins their business.  It's incredibly frustrating to see a company hamstrung by the "upgrade" they paid far to little for, thinking they were investing in their future I.T. capability and stability.  All of this happening because some sales person took it upon themselves to decide that what I spec out for the client is too expensive.

Here's some advice for sales and my fellow technical resources alike: ask the company for their "burn rate".  To explain, a burn rate is the amount of money that will be lost - per hour - if the IT infrastructure were completely lost; aka, network down.  This figure should include the cost of all labor that's now standing around not doing their job because they can't work since the entirety of their work is done on computer-backed systems (phones, email, wifi, internet, etc), along with the cost of any electrical, property, or other intermediate costs while down (paying rent on a building which isn't able to do anything, paying to keep the lights on, etc), and add that to the estimated number of sales lost during the outage, since sales can't do their work either.  They don't have to tell you this number - just know it themselves.  Fair warning: most don't know what this number is.  I have yet to find a company that goes back to the drawing board and figures it out during a sales discussion on the solution to buy; but it's an important thing to know as a business.

If the solution provided enhances their up-time/availability, and reduces outages from ~1% to 0.1% year over year, then that reduces their overall lost productivity from I.T. outages from 3.6 days, to 8.7 hours.  That's a lot of extra productivity, even losing two days at 8 hours a day, for an outage, is 16 hours of burn rate avoided.

I understand that estimating the down-time for a given solution is a bit of a minefield; but there's no doubt in my mind that higher-rated, higher-grade, better and more costly equipment from a reputable manufacturer, versus a cut-rate, budget solution from a sketchy division of a company or sketchy company, or by hacking together a solution that's a hodge podge of various interdependent systems, simply the former is more reliable.  Like the difference between buying an HP server, with NBD support, vs a custom-built "server" from off-the-shelf components, the HP server is known, predictable, and supported by a third party with the availability and components to repair any malfunction of the system in a day or less, while the custom-built server is unknown to anybody who didn't build it, and there is nobody that knows how to fix it immediately; to me that's the difference between a surgeon, armed with the knowledge and information of full x-ray, MRI, and other scans, before performing a laparoscopic, precision surgery, and just cutting open the patient in an exploratory fashion, trying to root around in their innards to find the problem.  The latter will do more damage, take longer under-the-knife, and have a longer recovery time, than that of the former, by a large margin.

Doctors don't go in blind, why should we?  Buy the HP server (or dell, or lenovo, or whatever brand); with support/warranty, and you basically have an expert, on-call, for when surgery is required, able to read the system logs and diagnostics to find the issue (troubleshooting - or in the case of our previous medical example, perform the required x-rays, MRI's, ultrasounds, whatever, to find the issue) then go in and precisely, and quickly fix the problem without even touching anything that's not broken.

This is just one of many, MANY examples.

To conclude this very long, very rant-y post, the perception that I.T. is a cost center is antiquated, and any business owner that still sees things this way is short sighted at best.  I.T. is the lifeblood of your business, and should be treated as a critical resource in the grand scheme of the operation of the business.  Anything less is irrational, short-sighted, and downright dangerous to the continuity of your company, your workers, their livelihoods, and yours alike, as well as a disservice to your clients and partners.  Pay your I.T. workers well, they keep the proverbial lights on when it comes to operations; listen to them when they raise concerns, pay attention and take it seriously.  We don't really gain anything or lose anything if the company operates smoothly.  We will find work even if your business goes under; and bluntly, most of us, don't care if you want to run the business into the I.T. stone age with bad gear.  At the end of the day, it's your company; we typically don't have any stake in whether or not you succeed.  Our only driving force for even suggesting improvement is because job hunting sucks.  Make our lives easier, and we'll reward you with years of dedicated service; loyalty notwithstanding.

Do the right thing for you, your company, your workers, your clients and your I.T. folks.