Today is the first day of school for Elder, and one that will be entirely online. She’s got afternoon sessions, which means that Younger will be looking to me as a playmate. I don’t think I’ll be getting much, if any work done at all. We’re lucky she got the afternoon spot though, cause the eight to noon spot would be even worse.
Yesterday morning I produced a Labor Day Breakfast for the local political party. It was in Zoom, and they called me about three days out to put it together. I managed to use OBS as a virtual camera, and was able to queue up a half dozen videos in OBS so that we weren’t struggling with it using Zoom’s sharing capabilities. I just set the scene in OBS, put the spotlight on my video stream and hit transition. Voila!
It went really well, but was really harrowing. There was probably close to two dozen speakers, including our Senator, several Representatives, as well as numerous state officials. I managed to keep it on track, except for some of the speeches. By the end I was playing people off, Improv-style. The whole thing was about two hours. One attendee remarked that it was the best-run Zoom meeting that they ever saw. There was over one hundred people on it at one point.
Then as soon as I was done with that I had to get ready for a pool party, which meant I had to run to the grocery store for last-minute shopping. I only picked up a few things, but the store was crowded. As soon as I got back Missus loaded the kids up and we went to pick up her mom and headed off to the party.
I didn’t get to enjoy it for long.
About an hour after we got there, I got a message from a political committee member that the website had been “hacked”. I pulled it up and was met with the bare directory listing for the WordPress site. Index.php was missing, so my first thought was that it was a failed upgrade. I tried to pull up my management console on my phone to restore a backup, but the most recent one was months old. Oops. So much for relaxing.
I’m not going to get into the details of the hack or the recovery, I think this was a simple case of credentials being leaked. There were too many people that had access to it, and the committee secretary was given the site admin credentials to use to post on the page. Yikes!
I cleared that situation up with them and urged everyone to check their antivirus. One of the committee chairs was running a Mac with nothing but Malwarebytes on it. I swear. I locked the site down as much as I could with free versions of Securi and Ninja Scanner, so I think things are cool for now. This is the second time this site has been hacked though, so I’ll have to keep an eye on it.
I just checked the login attempts on the site, two attempts from Rio De Janeiro overnight.
It’s been a little over one year since I started blogging in earnest. I’ve been taking a look at the archives from last July to see what I was writing about back then. When I started, I think I gave myself a three hundred word target, just to get in the habit. Today, these posts routinely run two to three times that length, and with some posts in excess of fifteen hundred words. The content of those early posts were more focused; I had the habit of writing a post for every book or magazine that I read, but today these posts are mostly journal exercises for the most part.
My most popular posts have been on technical issues, two about a WordPress hack and an Windows server issue seems to drive most of the traffic here. My exploration into Facebook’s Prophet machine learning tools gets another trickle. I’ve yet to find a focus for this blog beyond whatever strikes my fancy for the day, and I’m content to continue with it as is, making small adjustments as necessary. However, they say that no one ever got where they wanted to go without a plan, so some critical fascimilie of a plan might have to come together at some point if I want this to be a part of a long-term career strategy.
For now, it serves enough for it to be a place where I practice my writing muscle. If I write, I am therefore a writer, so it goes, and every day that I write the better I get. I’m closing in on three hundred posts here, including ones older than a year old. (This count doesn’t consider the archival posts that are monthly roll-ups from the previous incarnation of my WP database.) I’m hoping that by the time I reach five hundred I’ll be even better. We’ll see if the traffic to this blog increases along with it. Time will tell.
The kids have been incredibly difficult this morning. We all got up pretty much at the same time, and I was unable to get much done till after they left for their grandmother’s house. Younger has been especially sensitive this morning, but both of the girls seem intent on making a sport out of disobeying me. I was unable to get either of them to do their studies this morning, and at one point I had them both taking timeouts in the kitchen, which they made into a game where they tried to laugh at each other while I made lunch. I shouldn’t be mad but I did lose my temper briefly from having to repeat myself whilst being ignored repeatedly. Hopefully they’ll be better behaved when they come back.
I’ll admit that part of the reason for the discord here in the house is due to a text I got from my WordPress client basically firing me from the project. When we had set out, I thought I had made perfectly clear that this was going to be done quickly. I believe my exact words were something to the effect of being on the cheap and good areas on the project triangle, and that if we needed to move to the fast that they should let me know. As we entered the third month of our engagement, they let it be known that they were frustrated with the pace, and that I had expressed some doubts about my ability to deliver the project. I had expressed some frustrations about the work that I had inherited. This was mostly due to the amicable arrangement that we had started out on.
I think one of the major mistakes I made taking on this project was not properly scoping it and setting expectations. Another WP developer in my area charges twelve hundred dollars for a basic, four or five page WP site, and this project involved a major redesign and restructuring of an existing site. Easily a six month project at the rates I was charging. That obviously wouldn’t have flown if I had proposed that at the beginning.
I did identify several aspects of the redesign that I wasn’t going to be able to deliver on my own, mainly image assets. I was having a hard time gathering stock photography to match what they were asking me for. When I made this clear to the client, and told them that delivering everything I felt needed to be done within the accelerated timeline was going to be difficult, they told me that they had other developer resources that we could bring in. I said by all means.
This hasn’t been going quite the way I hoped it would turn out. In anticipation, I wrote up a project summary, invited the outside dev to my Basecamp, where I had all of the project notes and tasks, and spent several sessions building out a backlog of things that needed to be done. I told the dev, a PHP and Laravel dev from Pakistan, that I needed their assistance with one particular task: setting up the MemberPress plugin for us.
It doesn’t seem that any of that has even been considered. When I got the text, to the effect that development would proceed from scratch due to the difficulty in determining what I had done, I checked logs for the staging site and saw that no one besides myself had even logged into it. So something else appears to be going on. I suspect that besides the English language barrier, the outside dev might be more of a Laravel developer than a WordPress one. And I find it highly ironic they’re starting from scratch, when I literally spent two months trying to figure out what the last dev did.
I’m trying to tread a fine line here given that this engagement is with someone I consider to be a friend. We had gotten into some heated discussions about this, and you know the old saw about mixing business with pleasure. Still, my friend is enough of a intrepid entrepreneur that I considered this a baby step into what should be the start of a mutually profitable enterprise for both of us. When they broached the subject of terminating the arrangement with me a few weeks ago, I was so held by a sense of honor that I basically volunteered to finish the work for free. That’s why this morning’s message stung so much.
I replied back with as much tact as was possible given the cortisone flowing. I told them that the outside dev hadn’t even given a cursory look at what I had done, and I asked that they take another look at the progress I had made in the past few days before they pulled the trigger on a redesign. Further, I said, even if they did insist on moving forward with a new project, I intended to continue my development on the staging site until I was satisfied that I had fulfilled my promise to deliver the redesign and the membership features by the end of next week.
This project has taught me a lot already, both about WordPress development, but aslo about managing client expectations. I have got to spend more time focusing on the business side of the relationship, and establish some formal contracts and work blueprints so that expectations are better managed up front. For now, I’ve got about twenty hours of work left in the month in which to deliver and salvage this project. Failure is not an option, and neither is ruining this friendship.
Quit messing around with lesser staging processes and get the real deal.
I don’t mean to be too glowing or make this seem like some infomercial endorsement, but I do really think it saved me from having a heart attack the past couple days. I’ve been using InfiniteWP to manage most of my stable of WordPress sites, and it’s served me well for managing updates and backups, and is even handy for migrating websites from one host to another. It’s well worth the $120 or so that I paid for it a few months ago. It’s staging features aren’t really that great.
Part of the problem is that it only wants to install the staging site as a subfolder of the main site. It also makes a copy of the database on the production database, it just uses a different table prefix. I shouldn’t have to tell you why this is not great from a performance and quota standpoint. The other problem is that it doesn’t provide much information when things go wrong. Ideally, I want my staging sites in separate subdomains, but IWP just can’t do this. And the documentation is very mum about this. I have a support ticket open with them right now to figure out why I was unable to clone a particular client site, and to make sure that this paragraph is correct. What I can tell you is that I spent days trying to get a proper staging site setup for my client using IWP.
It’s not all their fault. I’m taking over a project that seems to have been abandoned by the original developer, and there were many problems with the site that may have contributed to the problems I’ve been having, as we shall see shortly. IWP has three staging options, on the original site, on my configured staging server, or custom FTP. I was able to clone the site to my custom staging server, but the theme didn’t operate properly. I believe this may have been a problem with hotlinked theme assets, I haven’t figured it out yet.
I literally spent days trying creating subdomains and updating DNS on the client site, and couldn’t figure out why IWP kept giving me “error: check your hostname” when I tried to update things. I figured it was a DNS propagation error between the server hosting my IWP and the client’s host. I usually only work on sites I host directly, but this was the first time I actually had to use the staging features. I was getting very anxious. I had wasted several days was already dealing with an irate client, and was starting to get a panicked feeling when working on the project.
So I decided to go another route and explore some other options. I read through several blog posts on WordPress staging sites, and one name that came up several times was WPStagecoach. And it was only $12 for a month, so I signed up for a trial and had the staging site up in less than an hour. No kidding.
The setup process was impressive. Getting the plugin installed and activated was pretty standard, and creating the staging site was very user friendly. It started off by scanning the site for large files, and found a backup archive, which it asked to exclude. Then it starting creating a tar file of the site to move to staging, and showed me a status percentage as it did so. This was very much needed considering IWP had been “working” for hours without so much as a log update. After the tar process was completed, I did get an error that the archive was missing files, and was asked whether I wanted to abort, retry, or “proceed fearlessly.” I retried, waited another five minutes, and got the same error, so I went ahead and pressed proceed. Another five minutes, and BAM. There was my staging site, and it looked perfect.
And one thing that really impressed me was that after the creation of the staging site, I was given a list of errors that WPS had found, mainly places where the site’s URL was hardcoded in the theme templates. These are likely why I had the rendering issues on my previous staging attempt. So now I have a list of files that I need to target, as hard coded URLs will play havoc with my development environment as well. And this feature really shows how WPStagecoach really shines as a specialized product.
WPS hosts the staging site on their own servers, giving each site their own subdomain. I got ten with my account, which is way more than I’m going to need anytime soon. So now I can proceed with the next step on this project, which is getting our MemberPress module up and running. Then I’ll be able to see if pushing changes back to the live site is as easy as creating it in the first place. If my experience so far is any indication, it’ll be a sinch.
Since I was able to get my Substack post out on time yesterday, my focus has turned to my client’s WordPress project. I almost used the word “consulting” for a moment, but I am way too much involved in the design of the site to call it consulting at this point. And I’m having a hard time. Cause despite my familiarity with the tech stack involved in web hosting and CMS systems, my design skills really haven’t moved past the early days of the web.
Just look at this blog, for example, it’s focused on content, I’m still using a basic WordPress theme, and it’s got no color whatsoever. Part of that was intentional, so that I could focus on content generation and not get hung up on design presentation, but it’s hamstrung me. Part of my spiel whenever I talk to people about these project is about the separation of content and presentation. I tend to focus on content, structuring data, changing it. Design is not really my strong suit. It’s much more of a creative art that I’ve struggled with.
When I was in my twenties, I used to love to play around with Photoshop, making designs with filters and creating awesome images. I’m pretty good with typography as well. When it comes time to laying out text and images, and putting together a color scheme on a web page though, I just don’t know where to start. And therein lies my current problem.
The client came to me with a website that had been fully designed and rolled out between 2016 and 2018. It didn’t look like it was up to the design standards, even of that era. I can’t even put my finger on why I know that, but it just does. The site was broken. The first thing I saw when logging into the admin console was that it had a huge mess of plugins, and I couldn’t make heads or tails of it. I needed to figure out what was essential and what we could get rid of. And there were over twenty five hundred registered accounts in the system. Most of them looked like spam. Sigh.
I never ever ever make changes to a client site without testing it in a staging site, so that was the first order of business. I use Infinite WordPress to manage my sites, so the first thing I did was installed that plugin to connect it to my management console, then I started a backup. After we had that under control I cloned the existing site to staging and got to work. It was a mess. There was a lot of content on the site, over two hundred pages and posts, but the site used custom posts types and fields, and had some sort of content controls that were supposed to paywall the content and was just broken. I spent what must have been a week just trying to inventory the plugins to figure out which ones we could disable to speed things up.
The only other production change that I allowed myself was for analytics. I checked the console a couple of days later and found that the site was getting tons of hits. Apparently the blog content had been indexed by Google, and a lot of people were getting referred to it. Converting those hits into leads was now the priority. The site was still broken though. The navigation didn’t work, so most visitors were falling off. Furthermore, while checking the contact forms to see what emails were being used, I discovered that email for the domain wasn’t working. At all.
I was able to get the email up and running, but it took days to figure out where the problem was and to move the DNS records over to my reseller account. Delving into the theme itself, it appeared that the designer had just cloned the WordPress TwentySixteen theme and started editing it directly. I’m not sure if WordPress had child themes back then, but they had just dumped a bunch of custom PHP functions right after the existing ones, and the template codes were full of SQL queries into the custom fields, which didn’t jive with what the plugin documentation called for. I really didn’t want to spend however much time combing through this code to refactor it. So I recommended to the client that we start from scratch.
I’ve got an Envato account that I subscribe to, my process just involves finding a good theme that fits the tone of what I’m going for, throwing it up on the site, and swapping the boilerplate content with custom copy. And I don’t like writing copy for clients. Or, I should say I’ve avoided doing that in the past. It starts coming close to marketing, which I’ve had an aversion to. I’m working on overcoming that aversion, at least for myself and my projects, but doing that level of work for clients requires a level of intimacy with the project/product that I haven’t had the time for. I’d rather outsource it, and that’s just what I did with the last project that I was involved with.
I’m not going to take that route this time, we’re going to plow through this and get it done, come hell or high water.
One of the problems that I’ve found working through this project is that I tend to try and shape my content to fit the boilerplate in the themes I use. It isn’t working. My client gave me a document with the content that they wanted on the page, and my brain seized up looking at it, trying to figure out where to match it to fit. It’s not going to work. I’m going to spend some more time today going over it again to see if I can provide some structure to it, then I’ve got a work session with the client tomorrow to really try and break it down. I don’t think that we’re going to be able to do this part in WordPress directly. I may need some mind mapping tools or something else to make it work.
Pressing up against the limits of my abilities is going to take some work. We’ve got to bootstrap this site into production to get things moving. Maybe then we can engage a professional to help tune things.
For now, that’s me, designer, developer, editor, admin. While I have some downtime today, I’m going to do some research and see if I can find any other resources to help refine the process.
So yesterday’s party was a hit, and we’ve been pretty productive for what should have been a lazy Sunday. Yesterday, I roasted a seven pound pork butt and made carnitas out of it, and also wound up making the birthday cake and frosting from scratch. Just call me “Daddy Crocker”. This morning everyone but the birthday girl slept in, and I’m writing this in the middle of the afternoon. I spent most of the day cleaning up the mess from yesterday, and making some changes to this website.
We’ve been getting lots of Google referral traffic for two posts, one for a WordPress hack, and another for an obscure Windows server error message that I wrote about. I’m getting enough of them that I figure I might test whether there’s any opportunity for clients, so I put a Boxzilla popup on the two pages containing a contact form. That was easy enough, but I ran into problems with my WordPress SMTP capabilities that required troubleshooting. I wound up switching to the Post SMTP plugin, since it’s free. So, any hits on the form should send me a notification SMS via my cell provider’s text message gateway.
Now, I wait.
I’ve got a lot to do today, so I’m going to wrap this up and continue working on my Substack, which I’m hoping to get out tomorrow. I’m also overdue on my WordPress project for my client, so hopefully I can manage to get both of those done before the day is over.
Maintaining privacy for your kids by running a private WordPress instance in your home network
Well, I may have finally lost my mind. I quit Facebook over a year ago, but one of the things that I do miss is the throwback posts that they pop up on your feed with pictures and posts from one or five years ago. It’s great for those “oh look how little Elder is in that picture!” kinds of moments. I don’t feel comfortable sharing pictures of my kids on my normie feed anymore, but I still want to do more than just have a folder sitting on my computer somewhere that gets looked at once in a blue moon. Plus, Elder is getting more involved with using the computer, and I wanted to give her a chance to express her creativity without the risk of letting her have a YouTube account. So I did the only thing any sensible tech dad would do. I set up an internal WordPress site for the family to use.
Setting up internal domain
I’m pretty proficient with Windows domain controllers, and manage a lot of contoso.local domains that aren’t externally routable. I decided that I wanted to do this, so that the site could be accessed only from our local network. That way we could easily access it from any of our personal devices, and can potentially allow friends and family to look at it when they visit and join our network.
Bind is the go-to DNS server for Ubuntu, so I started by installing and configuring it. However I quickly got lost in a maze of .conf files and StackExchange posts trying to get it to work, so I dumped it and installed dnsmasq instead. Dnsmasq relies on simple host files instead of the more complicated zone files that bind uses, which is more than enough for what I need at the house.
I setup my /etc/dnsmasq.conf file as follows using this guide:
# Don't forward plain names or non-routed addresses
# Use OpenDNS, not ISP's DNS router
# Replace second IP with your local interface
Then I setup my /etc/hosts file with the records I need, pointing to the downstairs server and my development workstation.
After saving changes, I need to restart dnsmasq: systemctl restart dnsmasq. From there I was able to validate the configuration on the server and external machines using nslookup. Once I was comfortable that things were working, I added my internal server’s IP to my router’s DHCP DNS scope and refreshing the client leases on a couple devices to make sure they would work.
Note about .local domains
I’ve never had issues with domains ending in .local among my Windows business networks, but Ubuntu may have a multicast DNS service called Avahi running, which hijacks anything FQDN ending in .local. Interestingly this service was missing off of the actual Ubuntu server install, which interfered with my troubleshooting. The simplest thing to do to get us up and running was just to change our internal domain from dahifi.local to dahifi.internal. Any other non-routable TLD should work as well.
Additionally, Ubuntu ships with resolved, a network name manager service. It runs a caching service at 127.0.0.53, and interfered with my troubleshooting as well. My internal domain kept getting routed to my ISP’s search page, until I ran sudo service systemd-resolved restart to clear the cache.
You’ll notice that the site that the blog is published to is identified by the VIRTUAL_HOST environment variable. In this case we’re still pointing to the “top level” domain, dahifi.internal, and not blog.dahifi.internal. This is due to issues we were having with subdomain resolution on the Nginx proxy, and is something we’ll have to work on later. Originally, we had our internal GitLab instance running on this host at port 80, and had to take it down for Nginx to work. My next step is to make sure that subhosts work properly, and then reconfigure GitLab and this blog to run under something like git.dahifi.internal.
One additional change that I needed to make was to change the default file size limit of 2M. Following along with this setup tutorial, I added - ./uploads.ini:/usr/local/etc/php/conf.d/uploads.ini line to the WordPress container, then added the following to the uploads.ini file:
Then I rebuilt the container with docker-compose down and up, making sure to specify my blog-compose.yml file.
I still encountered errors trying to upload. WordPress kept throwing
Unexpected response from the server. The file may have been uploaded successfully. Check in the Media Library or reload the page.
I couldn’t find any errors in WordPress’s Apache instance, and PHP looked fine. I eventually found a message in the Nginx container log: client intended to send too large body. It seems that Nginx has it’s own limits. I added client_max_body_size 64M this directly in the /etc/nginx/nginx.conf file, then reloaded it with service nginx reload. Problem solved! Not only can I upload files directly through the web interface, but I was also able to add the internal site to the WordPress app running on my phone, and can upload images directly from my phone.
Elder is already working on writing a story in it now, and I’m looking forward to using this site as an internal bulletin board for the family. Let’s see what happens!
Getting it together, turning an inadvertent alarm into opportunity.
It is going to be a long day.
My Eldest, bless her heart, was using my iPad yesterday to film herself and her sister play with Legos, and afterward started playing with the stopwatch. She somehow wound up setting the alarm on my device for 4:35AM, so I got roused way early this morning. I tried to go back to sleep, but I think I woke myself up too much trying to disable the alarm. Que sera, here I am.
I must have spent at least eight hours working on the WordPress project yesterday. I can’t remember the last time I was that focused on anything for that long. A good half of that was trying to figure out why the header slider I was working on wouldn’t show up on the page I was creating. Turns out it was a sidebar setting. Lesson learned: WordPress’s page template hierarchy is quite complicated.
My internal GitLab server is already proving it’s worth, and has proved the need for some reengineering of the way things are currently laid out around here. I don’t have the development database fully under source control, so I was doing most of my work yesterday through a Google Remote Desktop session. And last night I was forced to edit images on Photoshop, check them into to version control, then import them into WordPress on the guest VM on the same machine, because I never properly setup guest to host file sharing and network communication. Well, I have a task for today.
Today’s Thursday, so tonight is the night I’m supposed to write tomorrow’s Substack. I’m having a bit of anxiety since I don’t really know what I’m going to write about. Maybe something about improving oneself during this time of self-isolation, with some mini-reviews of some of the books I’ve read recently? I hope I get a better idea before tonight. I haven’t checked my subscription stats since last week, so maybe I’ll see if anyone’s responded.
I am also expecting a delivery in sometime today. I ordered a new mesh wifi router, and am looking forward to setting that up. The wifi here has been garbage for a long time, and it is time for my old Netgear to go. I went all out and got an ASUS ZenWifi AC with all the bells and whistles, so we should be good to go. I just have to promise myself that I won’t set it up until after I finish my Substack, otherwise I’ll wind up distracting myself.
And I am going to start brewing beer! I made some preparations to use my Mr. Beer kit yesterday, and we should be good to start today. I also discovered that I did not need to order extra bottles, since the rest I needed were actually inside the brewing vat. Alas, now I’ll have some special ones I can use for gifts or something. The vat is assembled, the water is chilling in the fridge, and we’ll be ready to do this one with the kids as a project for today.
I must say that removing Twitter off of my phone may turn out to be another one of those life-changing steps, like when I dumped Facebook. I took a look at it yesterday after posting the blog for the day, and then looked at it once briefly during the day. I didn’t even feel the urge to pull it up during the evening, so focused I was on working on the WordPress project. My phone has turned into this strange, useless device now. I pick it up and look at it out of habit, but nothing on it has the same draw that flipping open Twitter and scrolling through it. Certainly not LinkedIn, that’s for sure.
And it has been several days since I filled out a job application. I’ve been somewhat discouraged by the requirements of some of the positions that I’ve been looking at, and my lack of experience in certain areas. For example, Twilio had a whole slew of postings up over the area recently, and there were several that I thought I would be a good fit for. One of the application questions asked “give an example of something you built using Twilio”. Ummm…. well, I put an Airtable base together that used Zapier to blast text messages during one of my campaigns, but I don’t think that’s quite what they’re looking for. I’m probably selling myself short.
I think the problem is two-fold. One, finding out from Boss that my future with Zombie, LLC was mostly secure for the near term, and that I’m not in any danger of losing my job. That has definitely reduced the sense of urgency. But secondly, I’m really so focused on finishing this damn WordPress project that I don’t want to risk abandoning it. It sound stupid really, that I’m holding back on finding that $100,000 job because I think it might take me away from this project that might lead to recurrent income.
Well, now I have my orders for the day: fill out that application for Twilio; write tomorrow’s Substack; and reconfigure my development VM so I can see it on my local network. That’s on top of doing whatever’s on my list for Zombie, and taking care of the kids and house today.
I had trouble falling asleep last night, Younger crawled in our bed just as I was dozing off and kept squirming as I was falling asleep, so I slept in her bed. It faces East, so I woke up at five and tried to go back to sleep. I heard Elder up, so I got up and started the day. She’s sitting across the room from me, looking up “Valentine’s Day” gifts ideas for the boy in our quarantine bubble down the street. Her sister has been ribbing her about it for days now.
One of our Zombie, LLC clients wants help standing up an internal GitLab server. It got me thinking, so I went ahead and set up a GitLab docker instance on my downstairs Ubuntu server. I figure it’s good practice. Do the job you want has always been good practice, so setting it up was worth the time. Plus it only took about fifteen minutes. The main problem I ran into was an SSH conflict with the existing service on the host. And it doesn’t appear that modifying the config on an existing container requires stopping the Docker daemon, so I just deleted the container and started over. I’ll probably move SSH if I ever do a real deployment, but here at the house the HTTP functionality is enough.
There’s also the mail issue. I didn’t want to use the root account to setup my repos, but the workflow around new accounts wants to send an activation email. I tried installing sendmail on the host, but the password reset didn’t work. I doubt it will work without a publicly routable dynamic DNS entry back to it or SMTP services, which I don’t want to mess with right now. Thankfully I found a password change form in the admin interface that didn’t require knowing the old password and got up and running.
I am nowhere near as strong with my Linux management skills as I am with Windows, where everything is pre-packaged and is somewhat unified. I can stand up domain.local services lickety split, and have a library of PowerShell scripts to setup AD, DNS, DHCP services within a domain. I have never actually taken the time to set one up at home though, but that point may soon be approaching. I’ve been wanting to investigate the use of Ubuntu server as an alternative or supplement to Windows based AD services, but part of me is skeptical that such a setup is even viable for workstation authentication and services. But I digress. The point I’m trying to make here is that I’ve always been in awe of Unix sysadmins ever since I worked at an internet service provider back in the late 90’s and watched our systems guy pop in and out of terminal shells like a wizard. I’ve never felt adequate in that regard.
I made some good progress yesterday working on the WordPress project, and have started converting the client’s site over to the new theme. I’m going over the demo site, examining the Bakery build they’ve got set up, and recreating it using the client’s assets. This allows me to get a bit more familiar with the framework that the theme author is using, and hopefully gleam some best practices at the same time. It’s a two step forward, one step back process. There are some strange bugs that popped up. Activating Woocommerce seems to bring the site down completely, as does changing the theme back to the original. Then at one point, while I was working on the new header, the previews stopped working completely and would only throw 404 errors. They work in the actual site, so I had to make do while I made edits.
Usual best practices for WordPress development and git repos are to exclude the entire WordPress directory except for whatever theme and custom plugin that you’re developing, but since in this case we’re working on an entire site, I’ve added the entire WordPress directory and associated SQL database files. The wp-content/uploads directory is mounted outside the container, along with plugins and themes. I haven’t yet pulled this directory on another machine yet, so I don’t know if it’s going to work. My main concern is how I’m grabbing the database. Managing PostgreSQL during my Django projects has always been a bit of a pain as I never learned how to incorporate it into my source control. I’ll have to spend some time correcting this deficiency.
Here is a look at the Docker Compose file I am using for my development setup. The SQL mount /docker-entrypoint-initdb.d/backup_to_load.sql get’s imported when the container is created; I assume that it’s ignored when pulling the SQL data from source. We shall soon find out. Also, I haven’t solved the file permissions issues that happen when trying to edit things like the wp-config.php file. I’ll have to save that for a later time.
This attack is not new, but the tactics are evolving, and some people are still behind the curve
I’ve been managing business networks for some time, and I’ve witnessed phishing attacks, where attackers attempt to steal a victim’s email login information, evolve the last few years. Yesterday I was alerted to a new variation on this traditional attack that I thought was worth sharing and dissecting, as you’ll see why.
Almost all of the attacks that I’ve seen stem from an email that a victim receives. Usually it’s someone that the victim has corresponded with in the past. The subject line and body vary, but there’s usually an external link where the victim is directed to in order to download some secure file. Normally, the victim arrives at a page that looks like a Google or Microsoft landing page, but of course they’re a fake, setup to steal the victim’s credentials.
If the phishers are successful, they’ll have gained access not only to the victim’s mailbox, but also any associated document storage systems like Google Drive, or Microsoft OneDrive or SharePoint. From there it’s all over, the attackers can download whatever they need, or if they discover that they’ve infiltrated a high value target, they might lurk, and prepare additional attacks.
In one particular case that I was involved in a few years ago, attackers managed to phish the CEO of a company. They discovered that they were going to be travelling from the East coast to the West, and waited until they were thirty thousand feet in the air to launch a fake CEO attack, requesting that their finance director wire tens of thousands of dollars to the perpetrators bank account as soon as possible. In this case, there were enough red flags that the attack was thwarted, but not before the attackers had used the CEOs mailbox to resend the phishing attack to everyone in their contact history.
And so the cycle repeats.
How not to be a victim
Normally, there are numerous red flags when phishing attempts happen, but it still surprises me the number of requests I get from people asking me to inspect an email for legitimacy.
Sometimes it’s as easy as examining the email recipient, or the actual link in the email, and finding that they don’t match. If Jane Doe’s corporate email is email@example.com, and you see your email client only displays “Jane Doe“, you might need to hover your mouse over it to see that the email is really from a different address altogether. (Hover over the link above to see what I’m talking about.) Most modern email clients have updated the way they display emails, making sure that the actual address is “Jane Doe <firstname.lastname@example.org>” or something similar.
However, there are still a number of businesses that haven’t taken precautions to protect their own email systems from being spoofed. That’s to say, there may not be anything stopping from someone from setting up a rogue email server and sending an email from anyone at that company. There are several methods to protect from this, known as SPF, DKIM and DMARC, that protect from this happening, so you may want to make sure that your domains are protected.
The flag that I look for is where the link is pointing. Just like email addresses, these URLs can be spoofed. Modern rich-text or HTML mail clients which allow special formatting can be used to try and trick users with links that misdirect users to hacked sites. So always check the URL. That official looking login page for your Office365 account might just be a fake sitting behind someone’s hacked WordPress site. CHECK. THE. URL.
These tips alone should prevent most people from falling victim to one of these attacks. If I’ve been drawn into investigating at this point, I usually go a step further and try to get the fake landing page taken down. Sometimes it’s easy to find the company who’s site has been hijacked, and usually a courtesy call is enough for me to consider my good deed done for the day. Sometimes the site is set up by the hackers themselves. A ten dollar web domain with a three dollar hosting account, paired with a free WordPress template is enough to start with. In these latter cases, I have to do a bit more work to find where the domain is registered and where the site is hosted. Then, an email to the company’s abuse department, and I’m done.
How you can stop it
And in almost every case that I’ve seen, it’s been a WordPress site that has been hosting the fake landing page. As it’s the software behind more than a third of all websites on the internet, it’s not surprising. But if you’ve got a business website running on WordPress and you’re not maintaining it or paying someone to manage it for you, then not only are you exposing yourself, your firm, and your clients to hacks, but you’re also partially responsible for any victims that fall prey through your site. Update your site, at least quarterly, or purchase a product or hire a firm that can check it on a regular basis for you.
Making sure email the security protocols mentioned earlier, (SPF, DKIM and DMARC) are enabled on your domains will prevent hackers from faking your domain and using it in an attack.
Using updated email software and security applications are also an effective way to mitigate these attacks. Make sure that your email client software is a recent version, or use a cloud-based one to make sure that you have access to the latest anti-phishing tools. And make sure you use them! It still astonishes me how many small firms haven’t enabled two factor authentication for their employees, or even looked at the protection services that are available from their email providers.
And one of the most important things you can do is train your staff how not to fall victim to these attacks. There are a number of firms that can deploy phishing attempts against your staff, and provide training to those who fail to avoid it.
Attackers upping their game
What concerned me with the attack I witnessed was the way that the attackers changed their tactics to evade some of the more advanced mitigation techniques that are in place to stop these cybercrimes. A number of enterprise level email security services have the ability to filter out these malicious links and block them from the recipient. They usually rely on some sort of whitelist or blacklist to allow certain domains through. In the case this week, the victim was sent to Live.com, which is Microsoft’s ID portal for Outlook.com and OneDrive accounts. To the casual observer, it looked like a legitimate OneNote notebook, and there was no breach at this point. No doubt most organization administrators would have no problem with users going there.
Of course within this OneNote page was the real trap, a link to the fake landing page. Thankfully the mark in this case, noting that the OneNote page was addressed from a person different than the original email, was suspicious enough not to fall for it. That said, when I was alerted to it and took a look at the OneNote page without the context of the original email, my initial thought was that it was legit. I almost cleared it! A second read turned up some irregular grammar, which is when I noticed the external link and the O365 landing page. Even then I still had to look up the domain registration on the site, two months earlier using an Asian registrar, before I was convinced it wasn’t some sort of Single Sign On configuration.
Technology changes fast, and cybersecurity is a cat and mouse game between attackers and the security professionals that protect your personal and business assets from these dangerous breaches. If you need help with managing your infrastructure or mitigation strategy against these attempts, let’s discuss it. Whether it’s email and network infrastructure, securing your website, or doing mock infiltration testing or employee training. I can help.
Well I sure did step up my game. I guess I was anxious about writing a big long post for today’s Substack post, so I wrote it last night. Someone put out a message on the company Slack yesterday about a suspicious email, and I almost fell for it. I figured it warranted an exposition, so I spent an hour or so last night writing it up, after everyone went to bed. The experience writing at night, undisturbed, is quite different from my morning writing, when the kids are getting up and wandering in the room every five minutes. I’m not sure I like having a deadline every night like that though, so we’ll keep it limited to Thursday nights. I think I’ll even put it on my calendar just to keep it routine. Done.
I didn’t make any progress on my WordPress client last night, instead, I’ve started reading through the Docker documentation so that I can figure out what the hell I’m doing with my setups. I know a couple commands; it feels like I’m an infant failing my limbs trying to grab a rattle but I keep hitting myself in the head instead. I don’t understand half of the stuff I’m reading when I look at some of the image notes in Docker Hub, and trying to read through some of the discussions on GitHub issues is even worse. So I’m just reading through the best practices documentation to try and get a sense of how things run between Dockerfiles and Docker Compose, so that I can load a SQL backup automatically, add access my PHP files in both my development environment and in the container’s web services.
And the Docker documentation is pretty interesting. It’s scattered with little goodies like links to Twelve-Factor Apps, and here documents, which is something I didn’t even know about. And if there’s one place I need to step up, it’s my bash skills.
Anyways, I get to keep this post short since today’s a double post day. I thought about writing something for an off day, but figured the ritual of writing morning pages was more important than taking a day off. I’m starting to gather ideas faster, and my writing output has stepped up quite a bit. So I’m going to wrap it up, make some changes to today’s big post before I blast it out to several hundred unknowing Substack subscribers.