Stock price forecasting using FB’s Prophet

I’ve spent countless hours this past week working with Facebook’s forecasting library, Prophet. I’ve seen lots of crypto and stock price forecasting and prediction model tutorials on Medium over the past few months, but this one by Senthil E. focusing on Apple and Bitcoin prices got my attention, and I just finished putting together a Python file that takes his code and builds it into some reusable code.

Now after getting to know it better, I can say that it’s not the most sophisticated package out there. I don’t think it was intended for forecasting stock data, but it is fast and allows one to see trends. Plus I learned a lot using it, and had fun. So what more can you ask for?

As background, I’ve been working on refining the Value Averaging functions I wrote about last week and had been having some issues with the Pandas Datareader library’s integration with the Alpha Vantage stock history API. Senthil uses a different third-party module that doesn’t have any problems, so that was good.

Getting started

I ran into some dependency hell during my initial setup. I spawned a new pipenv virtual environment and installed Jupyter notebook, but a prompt-toolkit conflict led to some wasted time. I started by copying the original code into a notebook, and go that running first. I quickly started running into problems with the author’s code.

I did get frustrated at one point and setup Anaconda in a Docker container on another machine, but I was able to get my main development machine up and running. We’ll save Conda for another day!

Importing the libraries

Like most data science projects, this one relies on Pandas and matplotlib, and the Prophet library has some Plotly integration. The author had the unused wordcloud package in his code for some reason as well, and it’s not entirely certain how he’s using the seaborn module, since he doesn’t explain his plots. He also listed two different Alpha Vantage modules despite only using one. I believe he may have put the alphaVantageAPI module in first before switching to the more useable alpha_vantage one.

We eventually added pickle, dotenv, and bokeh modules, as we’ll see shortly, as well as os, time, datetime.

import os
import pickle
import time
from datetime import datetime
import pandas as pd
from fbprophet import Prophet
from fbprophet.plot import plot_plotly
from bokeh.plotting import figure, output_file, show, save

import matplotlib.pyplot as plt
# import plotly.offline as py
# import numpy as np
# import seaborn as sns
# from alphaVantageAPI.alphavantage import AlphaVantage

from alpha_vantage.timeseries import TimeSeries
from alpha_vantage.techindicators import TechIndicators
from dotenv import load_dotenv

Protect your keys!

One of the first modules that we add to every project nowadays is python-dotenv. I’ve really been trying to get disciplined about 12-factor applications, and since I’m committing all of my projects to Gitlab these days, I can be sure not to commit my API keys to a repo or post them in a gist on my blog!

Also, pipenv shell automatically loads .env files, which is another reason why you should be using them.

load_dotenv()
ALPHAVANTAGE_API_KEY = os.getenv('ALPHAVANTAGE_API_KEY')

Fetch Alpha Vantage data

If there is anything I hate, it’s repeating myself, or having to use the same block of code several times within a document. Now I don’t know whether the author kept the following two blocks of code in his story for illustrative purposes, or if this is just how he had it loaded in his notebook, but it gave me a complex.

from alpha_vantage.techindicators import TechIndicators
import matplotlib.pyplot as plt
ti = TechIndicators(key='<YES, he left his API KEY here!>',output_format='pandas')
data, meta_data = ti.get_sma(symbol='AAPL',interval='daily', time_period=60,series_type = 'close')
data.plot()
plt.show()

from alpha_vantage.techindicators import TechIndicators
import matplotlib.pyplot as plt
ti = TechIndicators(key='Youraccesskey',output_format='pandas')
data, meta_data = ti.get_rsi(symbol='AAPL',interval='daily', time_period=60,series_type = 'close')
data.plot()
plt.show()

If you look at the Alpha Vantage API documentation, there are separate endpoints for the time series and technical endpoints. The time series endpoint has different function calls for daily, weekly, monthly, &c.., and the Python module we’re using has separate methods for each. Same for the technical indicators. Now in the past, when I’ve tried to wrap APIs I would have had separate calls for each function, but I learned something about encapsulation lately and wanted to give something different a try.

Since the technical indicators functions share the same set of positional arguments, we can create a wrapper function where we pass the the name of the function (the indicator itself) that we want to get, the associated symbol, as well as any keyword arguments that we want to specify. We use the getattr method to find the class’s function by name, and pass on our variables using **kwargs.

def get_technical(indicator, symbol, **kwargs):
    function = getattr(ti, indicator)
    data, meta_data = function(symbol=symbol, **kwargs)
    print(meta_data)
    return data

sma = get_technical('get_sma', symbol, time_period=60)
rsi = get_technical('get_rsi', symbol, time_period=60)

Since most of the kwargs in the original were redundant, we only need to pass what we want to override. We’ve not really reduced the original code, to be honest, but we can customize what happens after in a way that we can be consistent, without having to write multiple functions for each function in the original module. Additionally, we can do this for the time series functions as well.

# BEFORE
from alpha_vantage.timeseries import TimeSeries
import matplotlib.pyplot as plt
ts = TimeSeries(key='Your Access Key',output_format='pandas')
apple, meta_data = ts.get_daily(symbol='AAPL',outputsize='full')
apple.head()

#AFTER
def get_time_series(time_series, symbol, **kwargs):
    function = getattr(ts, time_series)
    data, meta_data = function(symbol=symbol, **kwargs)
    print(meta_data)
    return data

ticker = get_time_series('get_daily', symbol, outputsize='full')

You can see that both functions are almost identical, except that the getattr call is passing the ts and ti classes. Ultimately, I’ll extend this to the cryptocurrencies endpoint as well, and be able to add any exception checking and debugging that I need for the entire module, and use one function for all of it.

Writing this code was one of those level-up moments when I got an idea and knew that I ultimately understood programming at an entirely different level than I had a few months ago.


Next, we’ll use Pickle to save and load our data, and start manipulating our dataframes before passing them to Prophet

Fast notes

Completion of three-day fast

I completed my three day fast last Thursday, and, as I recall from last time, the third day was much easier than the second. My main mistake was not carrying a huge bottle of water around with me the entire time. I drank tea several times during the first two days and it made me feel better each time, but I think the issue may have just been dehydration in general. Having something in the belly to keep the stomach from grumbling helped as well.

In general, I believe that I was in much worse condition when I started this fast. I’ve been drinking a lot of sugary drinks lately, and was probably closer to ketosis that I have been recently, so the problems I had those first two days were likely accountable to that factor.

Breaking my fast in the evening this time seems like it may have been a better for me in some respects, but I think I may try to time it for breakfast the next fast since it’s probably easier to finish the last few hours while sleeping instead of going the entire day and trying to kill those last few hours while preparing for the first meal in three days. In my case it was Pad Thai, dumplings and rolls, followed by everything I could get my hands on for the following hours. I’ve been scaling back into my time-restricted fasting schedule, and am going back on my regular 11AM-8PM schedule starting now.

100 days alcohol-free

Thursday was also 100 days alcohol-free for me, which is probably something that hasn’t happened since I was a teenager. I don’t know if this means I’m a teetotaler now or what. At this point it’s not about drinking so much as it is about spending my money on alcohol. It’s not like I threw out my liquor and wine — I still have a couple unopened bottles sitting on the hutch that I’m saving, but at this point the habit of not drinking has its own momentum. And although I haven’t thought about it too much, being on caffeine and staying up late working on code or whatever is a much better feeling the next morning than staying up drinking.

Fast notes

So we are almost 44 hours through our 72 hour fast, or about 60% of the way. I’ve been struggling a bit today, but I don’t think I’m feeling as bad as I did yesterday. I was very cranky and weak yesterday, but I don’t know how much of that was just due to the weather and the fact that my kids were being extremely obnoxious.

One thing weird happened this morning to me that is worth noting. When I was in the shower this morning and closed my eyes, I saw an afterimage. I stared at it, still with my eyes closed, and it took on a very distinctive shape, and became very vivid. But the strangest thing about it was that it retained its position in space as I moved my eyes around. Normally, when looking at an afterimage in physical space, it seems to move as the eyes move, but this did not. And the pure distinctiveness of the image was very remarkable. I could see it and look at it. I opened my eyes and closed them a few times to see what happened, and the effect persisted for some time.

Now, I’m pretty sure that no, I wasn’t having a stroke, but I was still pretty jarring. Some research suggests that afterimages are formed in the brain, not in the eyes, so what I experienced doesn’t make me think that there’s something wrong.

Other than that, I’ve been trying to keep my belly full with water. I had a couple energy drinks yesterday, which was a mistake; I had some heart pounding after the second one yesterday afternoon. Today I just had tea and coffee and things seem OK. Heart pounding after getting up or going up stairs seems to be similar to what I experienced last time. I had 500mg of melanin before bed last night and slept well, I’ll do the same tonight.

I don’t remember how I managed to do this last time with the kids around. I think I may have done it during the weekend. Part of my crankiness last night was due to the fact that my kids wouldn’t stay at the table and kept getting up or trying to bring food into the living room, which they know is against the rules. I understand why Dr. Peter Attia doesn’t like to fast around his kids. Nothing is more annoying than a small child crying that they’re starving less than two hours after you’ve served them dinner.

Housekeeping

So today marks the start of my second three-day fast. I scheduled this one right after I finished the first one, about three months ago, and I must say I’m a bit more nervous about this one than I was the first time. I think it’s probably because I feel like I’m a bit unprepared. I’ve been consuming a lot of caffeine and sugar lately, and I think I was a lot closer to a keto-friendly diet the first time I did this. We’ll see how things go.

I signed on a new client today. This one is a new landscaping company that my wife hired a couple weeks ago to do our yard. I’ll be setting up web and digital presence for them, putting together a branding and online marketing package. The other part of this will be helping them setup the operational tools as well. I think we’re going to do Jobber as it seems like the most features that they need: CRM, quoting, job management and invoicing. Pretty much the entire job lifecycle. Should be fun.

It also looks like I’ll be moving forward with setting up a Shopify site. I’ll probably try to sub this out, but we’ll see how much I think I can do on that.

I went ahead and pulled the trigger on Basecamp. It seems like the best of all the other project and client management software that I’ve seen. I’ll probably wind up having to pull Harvest into my stack as well just for timekeeping. The only other thing I’m missing is accounting. Right now I’m sending invoices through Paypal, but I’m going to need something more full featured. I know I don’t want Quickbooks. Freshbooks and Xero are the only two that I’m really aware of at the moment, so I’ll have to find something to use. Between the wife and I, we’re starting to take on a lot of additional work that will need tracking. I’ve been able to handle our tax returns via Turbotax Self Employed, but I don’t think it’s going to cut it much longer.

Washing C++ code

I spent most of the day hunched at my laptop, checking out git branches trying to rebase commits to clean up the project I’m working on, but I haven’t been having much success. I did, have some good progress with automated documentation and code review tools, as well as some Docker stuff.

I found an interesting presentation by a dev named Uilian Ries titled Creating C++ applications with Gitlab CI, which is exactly what I’m hoping to do. He mentions tools such as Cppcheck, Clang Tidy, and Doxygen. Now I remember something about automated documentation generators during one of my CS classes a couple semesters ago, but let me say that I really should have paid more attention.

Code dependencies for a wallet address creation test class in the Cryptonote codebase. Generated by the Doxygen automated documentation tool.

Uilian goes into a lot more during his presentation that I didn’t get into today, but I did start to work on automating the build process using Docker. One of the problems that I’ve run into with the original Cryptonote forks is that it was built for Ubuntu 16, using an older version of the Boost library. I haven’t quite figured out how to get the builds to work on Ubuntu 18, and keeping an older distro running somewhere isn’t really an effective use of time. I already had Docker setup on a home server, so I was able to spin a copy up, clone my repo, install build prereqs and go to town.

docker run --name pk_redux -it ubuntu:16.04
root@5a1d66905643:/#
apt-get update 
apt-get install git
git clone https://gitlab.com/pk_redux/pkcli.git
cd pkcli
apt install screen make cmake build-essential libboost-all-dev pkg-config libssl-dev libzmq3-dev libunbound-dev libsodium-dev libminiupnpc-dev libreadline6-dev libldns-dev
make
.build/release/src/pkdaemon

My next step here is to save these commands into a Dockerfile or docker-compose file that I can start building off of, adding the code checks and documentation generators as needed. Once I’ve verified the syntax and worked out any bugs, I should be able to start adding things to the Gitlab CI YML files as well. This should help keep the project well-maintained and clean.

I’ve been familiar with Docker for some time, but it’s been a long time since I last messed around with it. It’s really exciting, to be able to document everything, and be able to spin up containers without polluting my base system.

Becoming a Git-xpert

I have been trying to get a grip on the Pennykoin CLI code base for some time. One of the problems that I’ve had is that the original developer had a lot of false starts and stops, and there’s a lot of orphan branches like this:

Taken with GitKraken

If that wasn’t bad enough, at some point they decided to push the current code to a new repo, and lost the entire starting commit history. Whether this was intentional or not, I can’t say. It’s made it very tricky for me to backtrack through the history of the code and figure out where bugs were introduced. So problem number one that I’m dealing with is how to link these two repos together so that I have a complete history to search through.

Merging two branches

So we had two repos, which we’ll call pk_old and pk_new. I originally tried methods where I tried to merge the repos together using branches, but I either wound up with the old repo as the last commit, or with the new repo and none of the old history. I spent a lot of time going over my bash history file and playing with using my local directories as remote sources, deleting and starting over. Then I was able to find out that there was indeed a common commit between these two repos, and that all I had to do was add the old remote with the –tags option to pull in everything.

mkdir pk_redux
cd pk_refresh
git init
git remote add -f pk_new https://github.com/Pennykoin/Pennykoin-old.git --tags
git merge pk_new/master
git remote add -f pk_old https://github.com/Pennykoin/Pennykoin-old.git --tags

Now, I probably could have gotten away by just cloning the pk_new repo instead of initializing an empty directory and adding the remote, but we the end result should be the same. A quick check of the tags between the two original repos and my new one showed that everything was there.

The link between the two repos

Phantom branches

One of the things that we have to do as part of our pk_redux, as we’re calling it, is setup new repos that we actually have control over. This time around, everything will be setup properly as part of governance, so that I’m not the only one with keys to the kingdom in case I go missing. I want to take advantage of GitLab’s integrated CI/CD, as we’ve talked about before, so I setup a new group and pkcli repo. I pushed the code base up, and saw all the tags, but none of the branches were there.

The issue ultimately comes down to the fact that git branches are just pointers to a specific commit in a repository’s history. Git will pull the commits down from a remote as part of a fetch job, but not the pointers to those branches unless I physically checked them out. Only after I created these tracking branches on my local repo could I then push them to the new remote origin.

Fixing Pennykoin

So now that I’ve got a handle on this repo, my next step is to hunt some bugs. I’ll probably have to do some more work to try and de-orphan some of these early commits in the repo history, cause that will be instrumental in tracking down changes to the Cryptonote parameters. These changes are likely the cause for the boostrap issue that exists. And my other priority is figuring out if we can unlock the bugged coins. From there I’d like to implement a test suite, and make sure that there is are proper branching workflows for code changes.

Choosing when, and how to sell

Knowing when to sell is one of the problems I struggle with as a trader, both in equities and crypto markets. In the past, I’ve relied on what the Motley Fool has referred to as a ‘buy and hold forever’ strategy, and it’s worked out well for me with some of the bigger tech stocks. As someone who’s been focusing more on shorter time frames lately, I’ve been having less success. And after watching my crypto portfolio break six figures in the winter of 2017 before crashing almost ninety percent, I’ve been trying to find a way to ensure that I’m able to actually take some profits when my positions start taking those parabolic runs.

One of the interesting metrics around the USD price of Bitcoin is the Mayer Multiple, which is the multiple of the current BTC price over the 200-day moving average. Trace Mayer determined that, historically speaking, the best long-term strategy was to accumulate BTC when when the Mayer Multiple was under 2.4. For comparison, the last time we hit that level was late June of this year when BTC hit $14K. Now I have traditionally been one to dollar-cost average into BTC, weekly, but I had to stop my contributions for a variety of market and personal reasons, but one plan I have been thinking about is to sell a large share of my position when the MM hits 2.88. This is a number I cam up with just by looking at the charts, and is currently about $26,200.

So I was really interested by this strategy put together by former BlackRock portfolio manager Vishal Karir, on how to take profits before the next bitcoin recession. I’m not going to rehash the entire piece here, suffice to say it sets a static accumulation target, buys when the asset value is below this target, and sells when it’s above it. I wanted to do some backtesting with some of my assets and see what I came up with. So I wrote up the following code in a Google Collab doc so I could start playing around with it.

Instead of looking at BTC directly, I wanted to start by looking at the Grayscale BTC ETF, GBTC, so for this block we’re using Pandas wonderful datareader to pull quotes from AlphaVantage.

import os
from datetime import datetime
import pandas_datareader.data as web

!pip install ipywidgets
import ipywidgets as widgets
from ipywidgets import interact, interact_manual

os.environ['ALPHAVANTAGE_API_KEY'] = 'my_key'

f = ""

time_series = [
    ("Intraday Time Series", "av-intraday"),
    ("Daily Time Series", "av-daily"),
    ("Daily Time Series (Adjusted)", "av-daily-adjusted"), 
    ("Weekly Time Series", "av-weekly"), 
    ("Weekly Time Series (Adjusted)", "av-weekly-adjusted"),
    ("Monthly Time Series", "av-monthly"),
    ("Monthly Time Series (Adjusted)", "av-monthly-adjusted")
]  
     

def get_asset_data (ticker, time_frame="av-weekly", start_date="2017-01-01", end_date="2019-09-30"):
  global f
  f = web.DataReader(ticker,
                     time_frame, 
                     start = parse(start_date), 
                     end = parse(end_date))
  return f

interact_manual(get_asset_data, ticker="", time_frame=time_series)

We’re using f as a global for our dataframe. It stores the stock data. We’re also using ipywidgets to allow us to easily change parameters for the data we want to run against.


The next cell allows us to pass this previous dataframe to our backtest function. I wanted to play with various parameters such as the amount of capital available, the max contribution, and the ratio of max contributions to max sell amount.

def run_karir_target(price_data, capital, contribute=100, sell_factor=2):

  def get_asset_value(price):
    return (shares * price)
  
  def buy(amount, price):
    
    nonlocal shares, cash
    if amount > 0:
      if amount > contribute:
        amount = contribute
      action = "BUY"
    else:
      if amount < -sell:
        amount = -sell
      action = "SELL"


    num_shares = amount/price
    print("{} {} shares for {}".format(action, num_shares, amount))

    cash -= amount
    shares += num_shares

    
  f = price_data
  cash = capital
  contribute=contribute
  sell_factor = sell_factor
  
  shares = 0
  sell = sell_factor * contribute
  week = 1

  for i, j in f.iterrows(): 
    if cash < 0:
      print ("Out of cash!")
      break
    price = j['open']
    print("Week {}: {}".format(week,i))

    target = week * contribute
    value = get_asset_value(price)
    print("Target: {} Asset value = {}".format(target,value))

    difference = target - value
    # 0 - 500 = -500

    buy(difference, price)
    new_total = get_asset_value(price)
    total = cash + new_total

    print ("Total: {}, Cash: {}, Shares: {}, Asset Value: {}".format(total, cash, shares, new_total))  
    print()
    week += 1

Now there’s a lot here I will do later to clean up this code, making the params available via a widget as I did for the first function, but for now it works just fine. Here’s a test run with $100 contributed weekly from just after the beginning of the BTC bear market till now.

Week 1 and 2 of our backtest
The ‘bottom’.
We’re not including the entire run, but here’s where we end up.

So, not bad on our hypothetical run. That’s a gain of almost $4700 off of $3769 invested, or a 24% return. Now we’re cherry-picking, or course. At the bottom of the market, we would have had over $7K deployed at a break even. But Karir’s strategy opens up a whole slew of possibilities, and what may be a good rule of thumb for scaling out of positions. Since I’ve been able to get this package working, I’ve been looking at other investments that I’ve made in equities markets, and am starting to form some hypotheses that may help guide best practices for both entries and exits.

My next step, after cleaning up this code a bit, is to figure out a way to run some regression tests on the sell multiple. I think a dynamic variable may actually be more helpful in instances where the asset goes on a parabolic run. But the point here is that I have a framework that I can test my assumptions against.

It shouldn’t be too hard also to integrate Karir’s strategy for accumulation of BTC, as well as altcoin pairings as well. It’s an exciting strategy for investment, and one that can be automated via exchange and broker APIs. There may also be some variations that we can deploy, using this kind of targeted portfolio value as a way to layer limit orders. For this simulation we simply used the open price of the time period, but we could calculate the price an asset would have to be for us to sell it next week, and set some orders in the present period. If the high for that period hits that level, we could simulate the trade and see how that affects our gains.

I can’t wait to model this and put it to use.

Cryptoasset portfolio tools

I spent some time Sunday cleaning up all the old coding projects on one of my computers and uploading them to my GitLab page. Most of the repos are private, but I’ll talk about cryptomarketbot shortly. I had to go through everything to make sure all tokens and other identifiable information was moved out. I used the wonderful python_dotenv package for that.

Most of the packages related to campaign finance data, that will likely stay private until I’m ready to dox myself, or crypto or equities related stuff. There were a few commercial projects that I had parked on Bitbucket that have been moved. Now that I’ve been able to inventory what I have, I can start fleshing out the useful stuff a bit more and refining it to something useful for myself and others.

Cryptomarket Bot

Cryptomarket Bot, as I call it, is not particularly useful. I wanted to track the advance and declines in the top cryptoassests by market value, so I built a small function that queries the CoinMarketCap API and inventories the the top x coins and counts the number that have gone up or down. I think I had the idea while I was reading Alexander Elder’s book Trading For a Living, and it seemed easy enough to implement. I went and bundled that as a Twitter bot, but I haven’t been very motivated to maintain it. I suppose I should figure out a way to park it in a docker container where I can keep it running in-house, or push it to a Heroku hobby instance and leave it there. Maybe there are additional analyses that could be run, and the library could be triggered as a function call via a scheduler, cron or Celery, instead of a never ending Python script.

Finance libraries

I have a number of spreadsheets that I use to track my cryptoasset holdings. I have one for mining and masternodes, and a series of others that I use to track investments in alts that I’ve also started using to plan equities trades as well. The general idea follows Elder’s two percent rule, that no trade exposes you to more than two percent total portfolio loss. Calculating that number for a brokerage account is pretty simple, since everything is in cash or equities, but crypto is a whole other story. One has to determine whether portfolio value is going to be pegged against the dollar, or in terms of BTC, (I prefer the latter,) and many assets may have no direct pairings, such as new shitcoins that aren’t listed on exchanges, or ERC20 assets that are only pegged to Ether.

So while figuring out my risk profile for a ethbtc trade may be simple enough, determining that for something like IDEX staking is a bit more difficult. It’s also hard to separate long term buys, (dollar cost averaging BTC,) from more speculative plays like trying to swing-trade PIVX or something. I’ll be spending more time in this space walking through that decision-making process as I figure out ways to model my portfolio.

Tomorrow marks the start of the last quarter of 2019, and we’ll use the date to mark a new snapshot of my holdings, figure out what my strategy is for the quarter, and will walk through the trades as I plan them out, execute them, and track them. Stay tuned.

Continuous integration/deployment for static web sites

I spent most of my day yesterday setting up Jekyll on my local workstation and getting that setup. The group project that I’m working on at school requires us to publish a web page for it, but the course assignment requires that we only use static pages, so we can’t run WordPress or other CMS systems. The pages have to be easily transferable to a new location for posterity once the term ends, and it’s questionable if PHP is even running on the web server, and getting a MySQL database is a special request.

Anyways, there’s only one other person on our team besides me that knows anything about HTML. One of my teammates started threw a site together on Silex.me, which is a visual editor that allows you to download static files that can be thrown into the web server’s directory. Editing them isn’t so much fun. It requires downloading the files from the web server, uploading them to Silex via Dropbox, and then uploading the files back to the web server. Not efficient, and there’s no history.

Now our CS department does have a local instance of GitLab setup, which raises possibilities of using some continuous integration/continuous deployment automation to the process. (No Pages, though. Alas) After much debugging, we were able to create a job that would copy the HTML files from our repo’s public directory to the webserver’s secure_html location, where the web page is served from: .

image: alpine:latest 
before_script:  
 - apk update && apk add sshpass openssh-client rsync bash 

secure_html:   
 stage: deploy   
 script:    
  - eval $(ssh-agent -s)    
  - bash -c 'ssh-add <(echo "$SSH_PRIVATE_KEY")'    
  - mkdir "${HOME}/.ssh"    
  - echo "${SSH_HOST_KEY}" > "${HOME}/.ssh/known_hosts"    
  - rsync -auvz public/* user@hostname:/home/user/secure_html/   
artifacts:    
 paths:   
  - public   
 only: 
  - master 

There are a couple of of variables that we had to specify in our CI settings. SSH_PRIVATE_KEY is of course the key that we created and uploaded to the server to connect without a password. There should be no password on the key itself, as we could not determine a way to provide the password within the script. And it’s probably redundant anyways. The thing that caused us a lot of issues was figuring out that we needed the SSH_HOST_KEY to prevent a key verification error when running the rsync command.

This way, whenever someone modifies the HTML files in the repo, it will deploy those files to the web server without any intervention. It will also allow us change history, which is also crucial if someone messes up. Another benefit is that we have a record of commits from the various team members, so we can tell who is contributing.

Now since HTML still requires a bit of finesse to work, we’ve been converting our site over to Jekyll, which should allow us to use either Liquid or Markdown for our content. I was able to get a local development environment up and running and generate a basic template for our site, so now the next step involves deploying a job in CI that will build the site, then push over the static site files to our web server. We’ll cover that later.

Scaling a managed IT service provider

The company that I work at is coming up on seven years old this winter. We’re a small managed service provider with about 4 employees and 25 or so clients. We provide IT support and project implementation services for small professional and service companies. We’ve been stagnant, growth wise, for the past three years or so, and my main focus in addition to taking care of our clients is refining our business processes so that we can scale to the next level. What we’ve been doing has brought us success, but it’s not enough to get us to where we want to be.

We’re part of a franchise system of independent operators all over the U.S. The home office is supposed to provide us with best practices and partner relationships, and the franchisees pool their purchasing power to get best deals with the partners. That’s how it’s supposed to work, anyways. What’s happened in practice is that the home office basically provides new franchise owners with a vendor for this, a vendor for that, and so on, and basically leaves the franchisees to themselves to figure out how to implement it. It’s completely inefficient. I can’t even begin to tell you how much time we’ve spent managing our RMM and PSA tools, or how much of my day to day is refining these various systems (some of which don’t have any API for automation control) to talk to each other.

Instead of pooling human resources, say to have a team of engineers that specialize in setting up firewall systems, for example, each location pretty much has their own teams. We rely on outside NOC and helpdesk partners to deal with first-line issues, and the local teams are supposed to be escalation support. But providing information to these various entities can be very difficult (ITGlue has helped tremendously!) but having a remote helpdesk is very frustrating for customers who expect some sort of continuity.

Unfortunately we’re just not able to provide that level of service for what clients are willing to pay. Especially the smaller clients. MSPs use a per-month contract billing, with rates for servers, workstations, and other IT resources, but that usually just covers keeping things running, remotely, and on site and project work is billed separately.

Things can really add up for clients, especially when they don’t follow our recommendations and shit goes south. Most of them are trying to balance the cost of having their own in-house IT resource, but hardware, software and human resource costs can quickly add up. This is even more true when you consider regulatory and compliance requirements. It’s really hard.

And companies that skimp on these costs always pay for it. Always. I’ve had my fair share of ransomware breaches, but one that I saw this week really took the cake. An firm who we have done business with in the past, that we’ve been under a limited engagement with, had a really bad attack which took down their entire Windows domain: three servers, including AD, Exchange, SQL, file services, and a custom database application. We stopped doing business with them three years ago because it was always a challenge to justify what needed doing over there, and things were usually such a matter of urgency that we would be forced to do things to keep them running. And then we would have to spent weeks having to pull teeth to get paid. We finally said enough is enough and just walked away.

So we got a call from them a few weeks ago. Turns out they had pissed off another MSP, and needed help. They had been through several in-house IT resources, but they needed RMM monitoring, AV and patch management stuff that we would provide. But because they were in dispute with the old IT company, we weren’t able to get access to their backup and data continuity appliance.

Long story short, they got hit earlier this week and didn’t have backups for half their shit. I had convinced their in-house person that they really needed to get some sort of local backup, and thankfully they followed my advice. But it was really too little, and they’ve spent the last 72 hours trying to recover. And let me tell you, it was the most stress-free disaster recovery that I’ve ever dealt with. I’ve damn near had panic attacks and probably lost years off my life from the stress of dealing with my own share of these disasters. Sometimes they were self-inflicted, other times not. But since I wasn’t the one holding the bag, I was chill as fuck.

I’ve saw the writing on the wall for MSPs some time ago. I don’t know if it will be ten years or when, but the business model is going to approach a race to the bottom. And our local market is already saturated with 4 or 5 decent competitors, and many more not so decent. Internal conversations around the future of our firm talked a lot about compliance auditing for DOD/NIST, and the question we’re struggling with now is whether we want to be an MSP that does compliance, or a compliance firm that does MSP. My gut tells me to go where others aren’t. Which is why I’m focusing my time on process automation, combining applications via API.

I was able to list several things to our no list, things we’ve done in the past that have gotten us into trouble in the past. That means setting boundaries for business that we deal with, and will likely involve cutting some of our clients who aren’t growing with us or don’t see the value of the service we provide. It means converting our services to product offerings in order to differentiate ourselves from the competition. And it means automating our processing so we’re not making the same decision over and over again.