Automated IT support system

Pulling together a franchise system

I’ve been reading The 10% Entrepreneur, and The Future is Faster Than You Think in bed the past few nights, hoping that it’ll prime my brain to come up with ideas for me to take in the next step in my career. I’m not going to share the crazy dreams I had last night, but I will share this idea, mostly for me to mull on and come back to in a few months.

This idea is for a IT support communications app utilizing voice and chat, with a little bit of AI thrown in for speech recognition.

The problem: I work for an MSP, and over time a number of customers have gotten ahold of my personal cell phone. This is bad. We have several endpoints, if you will, for clients to contact us: an official office number, which is routed to an answering service, a support email, which goes to our ticketing system; and a support desk number, which goes to our helpdesk partner. We have no texting capability.

The catalyst: During a client network outage, I called their ISP, and the hold message said that I could text an agent at a certain shortcode. I quickly hung up and texted the number, then proceeded to deal with the issue, asynchronously. I thought that it would be a powerful tool for us to use if we had similar agent capabilites.

I’ve done a bit of tinkering with Twilio’s platform, having used it experimentally during a few political campaigns. Their Flex platform is geared toward call centers and support agents, and all of their services have APIs, allowing it to be connected to other systems. (This interview with their CEO is enlightening.)

Solution: Build out a phone/text response tree and use it to replace our answering service with speech to text message relay, and also provide text messaging capabilities for clients as well. There are a number of ways to integrate this with our legacy ticketing system API for ticket creation, or, for ticketing creation and status changes in the other direction.

There’s another opportunity here as well. The firm I work for is a franchise, and there is so much redundancy built into the business model. Every independent franchisee has their own instance of our ticketing system, and has to hire or train their own resources to work with all of our different vendors: our RMM, PSA partners, cloud services (e.g. O365 various services), disaster recovery/business continuity, and so on. I don’t even want to try and count the number of vendors that I have to deal with. Personally, I’m well suited to this type of jack-of-all-trades position, but I’m at the point now where I brisk at having to learn or deal with some new system that doesn’t have APIs or programmatical interfaces. And through my interactions with other franchisees and techs in our Slack, I can tell that some of them are less than capable of handling some of these projects.

The onboarding process for our location was very difficult. We were pretty much handed some tools and left to figure them out for ourselves. There’s been some improvements in how this is handled more recently, but one of our vendor onboarding documents was near fifty pages of step-by-step instructions and screenshots.

I’ve tried to set up some automations internally, and tried to get traction among the other franchisees, but the appetite just doesn’t seem to be there. I just don’t think the owner community is really thinking along the same lines as me, and this is one of the main reasons why I don’t think the firm is a good fit for me any longer. I approached one of the home office leaders about using some of the API work I’m doing to do some cross-franchise data mining, and got dismissed out of hand.

I think there’s a huge opportunity to consolidate some of these roles and operations across the franchise system. In fact, I think that it’s the only way that some of the smaller franchises are going to survive. That said, I think the way the business model works, and the way the franchise system has been sold to the franchisees will allow these improvements to be made.

The system I’ve described above should allow multiple franchise locations to share the same dispatching and messaging contacts, and allow messages to be routed to the proper client owner. I will share this idea within the global group, to see if anyone is interested funding development of such a system.

Windows 10 Provisioning packages FTW

My company has spent the last few weeks trying to prep customers for the death of Windows 7, which happened officially earlier this month. I’ve been less than happy with the amount of buy in that we were able to get from clients, most of whom are either too broke or too stubborn to deal with the expense. We’ve had a few that are taking things seriously and are upgrading their machines, so I’m preparing to do a lot of installs.

In the past, when I worked in Enterprise, we would do images for the various models of desktop and laptops that we deployed. It made sense because of the scale. My work in the SMB space doesn’t necessitate this type of operation, since deployments are sporadic and smaller in quantity. Plus there’s so much churn in the OEM hardware market that it just doesn’t make sense.

As a managed service provider (MSP) with dozens of clients, I’ve been trying to standardize our operations as much as possible, but it doesn’t scale very well. We have a Remote Monitoring & Management tool (RMM) that we deploy to all our endpoints that installs our remote access and security tools, but we wind up with different installers for each site. We can create scripts to deploy some software, but it’s clunky and I don’t like using it. I’ve had some success deploying things like Chrome and Acrobat Reader, but it’s useless for some of the more obscure line of business vendors that haven’t packaged their installers for silent installations.

A majority of our clients are either on Windows Active Directory domains or Office 365, which uses Azure AD, so I’ve managed to write and collect a number of Powershell scripts to repeat common tasks, but even after seven years I haven’t been able to reach the level of automation that I’d like to be at. I’ve written about my attempts to integrate some of our platform APIs, but doing things at the user level is really difficult when you’re dealing with small sites of five or ten users.

Recently I extracted the installer from our RMM provider, and found that the package is just another executable with a settings file. One of the lines in this settings file contains our client’s unique ID in their system, and discovered I could use this as the base for a universal installer. I wrote a PS script to search and replace this string based on a hash map, and I even added a little popup selection box to run it. It wasn’t anything fancy, but it made my life just a little bit easier.

One of the things that’s always been a pain in the ass is dealing with the ‘out of the box experience’ (OOBE) that Windows presents the first time you turn it on. We’ve got a standard operating procedure for naming the default admin account and password, machine names, but for some reason it still gets screwed up. So I wrote another small script that I can run in Audit mode that imports an XML file to skip OOBE, create the account, and install the RMM tool. Life easier, but still buggy.

Lately I’ve been playing with Windows Configuration Designer. It creates Provisioning Packages (PPKG), which can be used by end users to do a lot of these things. It’s got some useful features: device naming; enroll in AD or Azure AD; local admin account; and adding applications. You can even specify WLAN authentication for wireless deployment. Unfortunately, it’s not a panacea as debugging packages (especially application installs) is a pain. One, WCD is buggy. The advanced editor started acting buggy when I started changing the application install order, garbling friendly and file names in the XML and finally throwing errors during compile time that forced me to start from scratch. And if the package installation fails, it can’t be run again.

I made the mistake of trying to wing the installation of some of the applications. It’s really good about MSI packages, but you better have your command switches right for everything else. I kept running into issues with Acrobat Reader. Apparently it was still throwing up the GUI, waiting for a finish, which caused the PPKG to stall. And after restarting the machine and trying to run the PPKG again throws an obscure error message that is not well documented. And don’t even think about running the package in audit mode. It won’t skip OOBE, and seemed to undo the Azure AD join that I did.

I wound up splitting the device and account setup into a separate package that I could rerun while I troubleshot the application installations through the main package. Eventually I started using a VM workstation that I could restore a snapshot on, but it was only seven laptops that I needed to deploy. I finally had a working package by the time I got to the seventh!

I’m starting to see the larger picture for a program that can edit the XML to dynamically generate provisioning packages for all of our clients. Grabbing the Azure token might take more time to work out, but I just need to swap out a few variables for device name, local admin, and could individually select applications to install.

One last thing about Azure AD, apparently, joining a Windows 10 device to Azure causes Bitlocker to be enabled. Decryption keys are synced to the cloud, which is a nice feature.

API obsession

I have been obsessed with APIs lately. Obsessed. Part of this stems from the interest in coding, of course, but part of it has come from a new focus on automating a lot of manual processes out of existence. I think I first really started messing around with them via crypto — of course — through the need to maintain price tracking sheets for my spec mining projects. I wanted to be able to keep track of the amount of coins that we were mining, the current price of said assets, and use that to calculate earnings and so forth. When I started tracking, I would manually get the prices from the exchange, paste them into a Google Drive doc, then copy my totals from one tab into a running monthly sheet. It quickly became tiresome, and when I found an add-on that someone had created to do lookups via CoinMarketCap (CMC), I became very interested in figuring out how it was done.

Eventually, I got interested in projects that weren’t available via this CMC interface, and had to start rolling my own. I was able to write Google scripts that could call the APIs of various exchanges and mining pools, to give me exchange totals, prices, and mining payouts. I’ve added them to a hodge-podge collections of scripts that I maintain in a sheet, so I can keep track of the entire venture. I use them to plan trades and track positions afterward. Of course Google Sheets has its limitations, and most of my work is in Python, but the basic premise is the same. Wrap an API request in a function wrapper, do something interesting with the result.

A lot of the interest also comes from my interest in automation. I’ve read the stories about people who have automated their jobs using Python, for example, and one of the fun things about APIs is that not only can you get information out of them, but you can send requests to them and make them do things for you. To stick with fintech for a bit longer, trade execution platforms are a perfect example of this. Being able to send orders to a trading platform through an API has enabled the high-frequency trading and bots to take over the markets. But my main interest is a bit closer to home, or work, to be more precise.

At my day job, we use several different systems to maintain our operation. The crux of it is a professional services automation (PSA) ticketing system and a remote monitoring and management (RMM) system. The two vendors that we use are integrated petty well. There are several major players in the space, and most of them plug together pretty well. The main issue is that the PSA requires a lot of manual setup and steps to do basic things like setting up new clients, configuring contracts, maintaining inventory. All which require multiple steps through their rather clunky UI. It’s a pain. Even something as simple as closing a ticket requires 4-5 mouse clicks.

Using the PSA’s API, I’ve begun to draft a collection of function that will allow me to close a ticket using a simple close_ticket(ticketID) call. I’ve developed more complicated functions that will create contracts, add products to those contracts and link assets from the RMM to those contracts. Right now I’m focused on standardizing operations across our clients, but there’s further opportunity to standardize operations between all of our franchise partners.

But perhaps the most critical opportunity that I’m focusing on within my day job is eliminating failures caused by human error.

Hotel Alexa

I’m staying in a hotel tonight. It’s not something that I usually do but there’s been a couple of family events that have brought me and the crew out of town. I woke up this morning before dawn and found myself staring at this bright green light a few inches away from my face. It’s probably just the thermostat or something, but it got me thinking: how long before Hilton and the other big hotel chains start putting voice assistants in rooms? 

I probably wouldn’t be thinking this if the front desk had picked up the phone any of the dozen times I tried to call them last night, but it seems inevitable that there’s some company out there working on a bot for one of these chains: “Alexa, I need more towels.” “Alexa, how do I connect to wifi?” “Alexa, what channel is Disney channel?” “Alexa, have room service bring me dinner.”


Seems inevitable. I’m sure it will be a while before people are willing to accept these things in their rooms. Some, like myself, aren’t comfortable having these things in my house, let alone rooms where they are sleeping or doing other more intimate activities. I assume that the hotels could hand them out at the front desk as an option: “Would you like a virtual assistant with your room, sir?” And then I’m sure it’d be a matter of time before they’d become standard deployments the way wifi repeater seem to be everywhere. 

Update: It seems I’m a bit late to the party. Amazon released their Alexa for Hospitality in June of 2018:

Guests will be able to do things like order room service, request a housekeeping visit, or adjust room controls (thermostat, blinds, lights, etc.) using an Echo in their room. They can also ask location-specific questions such as what time the hotel pool closes or where the fitness center is.

Some upscale Vegas hotel apparently pioneered Alexas in their rooms back in December 2016, and in October of last year, Marriot announced plans to run a trial in Charlotte hotels. One thing that we didn’t anticipate when we were writing this was the response of hotel staff, who saw these devices as a threat to job security as far back as September. It was apparently part of a list of concerns when  Marriott employees went on strike last fall.