Posted on

Fair Open Source

Last night I had the pleasure of meeting Travis Oliphant, one of the primary creators of Numpy and and founder of Anaconda. He’s currently the CEO of OpenTeams, a company attempting to change the relationship between open source software and the companies that build on top of it. I found out about the lecture and was interested in it because of an article I had read in Wired about technology’s free rider problem, and went to the event without knowing anything much at all about Mr. Oliphant. I soon found out who he was and was very grateful that I had come. I’ve spent a lot of time using Numpy, and I’ll admit I was a bit starstruck.

Travis’s lecture spawned from his experience working on Numpy. He basically gave up tenure track at Brigham Young University to work on it, and had to find other ways to support his family for the two years that he was working on the initial release. As was noted elsewhere, much of the tech boom over the past 20 years has been built on top of the contributions of FOSS developers like Travis and others. He’s a big believer of profit, and thinks that the lack of financial incentives in the FOSS space has caused several problems, including developer to burnout, leading to a lack of proper maintenance of these projects. Many of these projects, like Numpy, have become crucially important to the scientific and business community.

Tim Oliphant’s Pycon 2019 Lighting Talk about Quansight

Oliphant’s goal is to make open source sustainable. Quansight is a venture fund for companies that rely on OSS, one of the ones they’ve funded is a public benefit corporation called FairOSS, which hopes to support OSS developers through contributions from companies that use OSS. He’s also doing something very similar with OpenTeams, hoping to follow Red Hat’s model of supporting Open Source by providing support contracts for various projects.

These are all very worthy goals, and I was both impressed and inspired by his talk. It’s opened up some interesting career opportunities. I recently took my first developer payment through GitCoin recently, and it was a bit of a rush. Getting paid to work on Open Source Software seems like an awesome opportunity, and I’ll be keeping an eye on this for potential post-graduate plans.

Posted on

Thanks to supporters

I hate to use the term ‘life changing’ to describe what’s happened lately, but I’ve seen a few orders roll in for the EICAR products that I put together and it has really made my week. I’ve been dreaming of leaving the rat race and having some sort of self-sustaining revenue streams to help me escape wage-slavery or whatever you want to call it. I’m not being very eloquent right now, but the plan is to escape the need to trade time for money, and seeing these orders come through and be fulfilled without any additional effort on my part seems like a dream come true.

Love him or hate him, Tim Ferriss has been a huge influence on me. I read the Four Hour Workweek when it first came out, but for whatever reason I’d never been able to take that leap and to full bore into something that could do that type of revenue-generation. About the closest I’ve ever come to it has been investing, whether buying stocks or crypto and seeing things take off of their own accord. (The 15% rally in BTC this past week is no doubt affecting my mood.)

I’ll be honest, I won’t be paying the grocery bill with the proceeds from the sales that I did, but this past week is very inspiring and helps reinforce that another way is possible.

Posted on

Automating value average stock investing

I spent most of the winter break working on automating a value averaging algorithm that I wrote about several months ago. Back in October we started scaling into three positions that we identified based on our work with some predictions we did using Facebook’s Prophet earlier. My goal was to develop a protocol and work out any kinks in the process manually while I worked on building out code that would eventually take over. While I’m not ready to release the modules to the public yet, I have managed to get the general order calculation and order placement up and running.

To start, I setup a Google Sheet with the details of each position: start date, number of days to run, and the total amount to invest. I used Alexander Elder’s Two Percent Rule, as usual to come up with this number. Essentially each position would be small enough that I wouldn’t need to setup stop losses. From there, the sheet would keep track of the number of business days (as a proxy for trading days) and would compute the target position size for that day. I would update a cell with the current instrument price, and the sheet would compute whether my asset holding was above or below the target, and calculate the buy or sell quantities accordingly.

After market open, I would update the price for each stock and put in the orders for each position. This took a few minutes each day, and became part of my morning routine over the past two months or so. Ideally, this process should have only taken five minutes out of my day, but we ran into some challenges due to the decisions we made that required us to rework things and audit our order history several times.

The first of these was based around the type of orders we placed. I decided that I didn’t want to market buy everything, and instead put ‘good-until-cancelled’ limit orders in. When there was no spread between the bid and the ask, I would just match whichever end I was on, and if there was a split I would put my order price one penny in the spread. As a result, some orders would go unfilled, and required some overly complicated spreadsheet calculations to keep track of which orders were filled, what my actual number of shares was ‘supposed’ to be, and so on. I also started using a prorated target, based on the number of days with actual filled orders. This became a problem to track. Also, some days there were large spreads, and my buy orders were way lower than anything that would get filled. There were times when the price fell for a few days and picked up some of these, but keeping track of these filled/unfilled orders was a huge pain in the butt.

One of the reasons that it took me so long to develop a working product was due to the challenges I had with existing Python support for my brokerage. The only feasible module that I could find on Pypi had basic functionality and required a lot of work. It had no order-placing capabilities, so I had to write those. I also got lost working through Ameritrade’s non-compliant schema definitions, and I almost gave up hope entirely when I found out that they were getting bought out. The module still has a lot of improvements needed before it can be run in a completely automated manner, but more on that later.

So far I’ve got just under a thousand lines of code — not as many tests as I should have written — that allows me to process a list of positions, tuples with stock ticker, days to run, start date, and total capital to invest. It calculates the ideal target, gets the current value of the position, and then calculates the difference and number of shares to buy or sell. It then places the order. I’m still manually keeping an eye on things and tracking my orders in the sheet as I’ve been doing, but there’s too much of a discrepancy between the Python algorithm and my spreadsheet. I don’t anticipate trying to wade through my transaction history to try to program around all of the mistakes and adjustments that I made during the development process. I’ll just have to live without the prorated targets for the time being.

I think priorities for the next few commits will be improving the brokerage module. Right now it requires Chromedriver to generate the authentication tokens; this can be done using straight up request sessions. There’s also no error checking; session expiration is a common problem and I had to write a function to use to refresh it without reauthentication. So first priority will be getting the the order placement calls and token handling improvements put in and a PR back into the main module.

From there, I’d like to clean up the Quicktype-generated objects and get them moved over to the brokerage package where they belong. I don’t know that most people are going to want to use Python objects instead or dictionaries, but I put enough work into it that I want it out there.

Lastly, I’ll need to figure out how to separate any of the broker-specific function calls from the value averaging functions. Right now it’s too intertwined to be used for anything other than my brokerage, so I’ll see about getting it generalized in such a way that it can be used with Tensortrade or other algorithmic trading platforms.

I’m not sure how much of this I can get done over the spring. Classes for my final semester at school start next Monday, and it will be May before I’m done with classes. But I will keep posting updates.

Posted on

Will this image break automatic camera scanner systems?

EICAR antivirus test string

The image above is a QR-encoded representation of the EICAR antivirus test string. The string (`X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*`) is used to check the effectiveness of antivirus systems without using actual malware. It was brought to my attention via a Tweet by computer security Rob Rosenberger in which he placed QR code on his car during a recent road trip. The intention here being to get automated scanners to read and insert the string into their databases, hopefully causing them to crash.

A few people mentioned how they wanted these on hats or shirts, so I spent some time today putting a couple of products up on the ecommerce site. After I put this out, someone noted that this string only works if it is the only data in a file, which reduces the likelihood that it will actually cause any havok.

Still, if the idea of trashing facial recognition systems or license plate scanners sounds like fun to you, then perhaps you’ll be interested in this.

Posted on

2020 Goals

I’m 40 years old, which means that I’ve spent half of my life in the 20th century and half of it in the 21st. Perhaps that’s why I wasn’t excited with the fact that we last night we moved into a new decade. I was reading a book last night when midnight came; my family was asleep. There had been pops of fireworks going off before midnight, but there was a point when they started firing off continuously when I knew that the New Year had come. I wasn’t sure whether it was the City or the nearby military base where they were going off, but after about what was likely the official display ceased, what sounded like small arms fire kept going on for another twenty or fourty minutes. I imagined handguns or semiautomatic rifles being fired off in the air. (I once found a spent 9mm bullet laying on my porch step, no idea where it came from.) The explosions were punctuated with the sound of sirens as emergency personnel responded, either to the illegal fireworks, firearms, or perhaps to some actual injury.

  1. Finish my degree: I’m one semester away from getting my bachelors in computer science. I’ll likely finish with a >3.5 GPA. This should open up some opportunities, but the worst thing about it will be that my loans will come due. This will likely cause some financial hardship until I accomplish #2…
  2. Get a raise: I’ve been at the same job and salary for four or five years now. My wife outearns me by a great deal, and it’s not a problem, pride-wise, as much as it is a difference in lifestyle. My budget is extremely tight and is a subject of stress, while she seems to have a lot more discrecion in how she handles her finance. I know that even with my lack of anything more than an associates degree I’m underpaid, but the freedom I have with my current job makes up for it. That freedom has downsides though, and it’s time to step up. Unfortunatley, I’ve come to recognize that my current job is with a zombie firm, which is why item #3 is…
  3. Find a new job: I’ve been with my current job for 7 years, which is longer than I’ve ever held a position anywhere else. I tell my wife I’m unfireable because of key-man risk, but I’ve come to realize that everyone else on my small team is as well, and that’s not a good thing. I’ll have to expand on this later. I have no desire to go back to a corporate position where I have to work more than 30 hours a week, so I will have to make sure to focus on #4…
  4. Pickup more freelance/contracting work: Until the next Bitcoin bull run makes me independently wealthy, I’m going to have to focus on expanding my independent work and pick up some actual clients. I just need to pick up sixteen clients at $250/month retainer to replace my current real job, so I’ll just need to focus on picking up a few more and staying on top of things while I work on a few more large development projects.
  5. Get my daughter sleeping in her bed: She’s three years old now, and still can’t fall asleep on her own. I don’t know if we broke the first one doing it, but at least she goes to bed without having to lay down next to her for twenty minutes. This will be a long hard fought battle, especially until I can get my wife on board.
  6. Deal with my defiant child: I have two daughters. My boss has two as well, and often tells me that I have no idea what kind of shit I’ll have to deal with when they become teenagers. He assumes they’ll live that long. I would suspect that my eldest has oppositional defiant disorder save for the fact that she’s an angel with everyone but my wife and I, so I just assume it’s our lack of parenting skills. I’ve got some books that I’m reading and some strategies that I’ll be working through, so we’ll see how that goes.
  7. Continue to meditate: Such a simple thing to do that has such a good impact on my well-being. I don’t know that I’ll be doing hour-long sessions, but twenty minutes a day seems like such a no-brainer. Figuring out how to make it part of my daily routine and get the kids involved will be key.
  8. Exercise regularly: I’ve been through so many phases on this that it’s ridiculous. I’m really going to need to prioritize this, along with meditation, to make sure that it gets done on a regular basis. When I’ve done so in the past, the results have been tremendous, but it’s been so hit or miss lately.
  9. The drink: the hundred-plus days off the wagon that I did late last summer was one of the best things I’ve done for my health. I’ve already accepted the fact that I’m an alcoholic (an addict might be a better description,) so I have to figure out whether I want to continue being a functioning alcoholic or what.
  10. Up my piano skills: music has always been part of my life, and I’m finally getting to the point where I can sight read. Besides a few pop tunes that I’ve been playing around with, I’ve almost mastered a Bach minuet. I really want my kids to pick this up by my example, but I’m having difficulty figuring out what an appropriate goal for me would be. Bach’s Inventions should be something I can achieve this year.
  11. Write: I am going to start a 1000-day goal to write at least 200 words to this blog.

Wish me luck!

Posted on

Parental payback

I’m not sure whether it’s seasonal affective disorder or just the holidays, but I’ve had a bit of ennui lately and have had trouble keeping up with things. I know it all stems from bad behaviors tending toward staying up late which has just been screwing with things the next morning, interfering with my good morning habits. I’m sure I could come up with lots of excuses as to why I didn’t exercise or have been consuming lots of sugar, caffeine and alcohol; staying up late on screens and so forth, but I’m not going to bother justifying it.

One of the thing that I’ve learned from meditation is the illusion of the self, specifically the storytelling that we all do to ourselves throughout the day to try to make sense of our lives. Our post-hoc justification for the way things are the way they are, or our apparent lack of free will. It’s so easy to fall back into old traps, old habits, and living with the cognitive dissonance between the way we are and the way we want to be can be disorienting if we pay too much attention to it, so we find ways to justify the way things are.

When I was younger, in my adolescence, I used to have these unhealthy behaviors with relationships — for whatever passed for relationships back in high school. Part of it was an inability to communicate, or even be able to acknowledge what I wanted, let alone express it when it came to another person. Eventually these unrequited feelings poisoned the relationship, a pattern that played out time after time until later in my adult life. I still struggle with it in my marriage now. But a few experiences opening up over the past few years have proved that a lot of the fear of acceptance that I might have felt were unfounded. It seems silly to say it as someone who has been married for almost ten years, or almost sad to thing that feelings of self-worth that I formed as a child are still affecting my ability to be happy as an adult. It just emphasizes the huge importance of being a parent.

Ultimately, I feel like I’m failing in that respect in some ways as well. It’s easy to slip into detrimental patterns in response to the way my children act, and it’s tiring to pay them the proper type of attention that they need. I have this need for authority in my household, for my children to obey and help with things like setting and clearing the table, doing laundry, and so forth. My children are so young, though, I wonder whether I’m pushing them too hard, but I always suspect that I’m not pushing them enough. I could tell stories for hours about my own experiences growing up, and I’m just emulating the behaviors that my father expressed, even though I have plenty of first-hand experience with how that backfired.

Or did it?

I won’t say that my dad was abusive. We get along fine these days. He just has a lack of, how do you say, couth, in many respects that I’ve managed to escape, thankfully. But he no doubt grew up in a much different world than today, and his daddy’s method of discipline would no doubt be considered child abuse today. Somehow, though, I feel like I’m failing as a father whenever words and reason fail me and I have to resort to physical discipline. But there are some times when my child just will not listen, becomes belligerent, and it feels like backing down is the wrong thing to.

My wife is a bit of a trained professional when it comes to kids. She’s worked in daycares and counseling with a background in early childhood development. I didn’t know shit about kids until I had one. She tries to tell me what’s ‘appropriate’ behavior for their ages, but even she resorts to less than ideal responses when our kids are being particularly difficult.

When I was younger, back home in the rural county where I grew up, I was around my cousins almost all of the time. We used to take great pleasure in driving the adults around us completely insane. Acting up to see how we we could push the limit, driving the adults around us to cursing and yelling. We thought it was a game.

Turns out payback is a bitch.

Posted on

Windows Feature Installation The Referenced Assembly Could Not Be Found. Error: 0x80073701

My day to day involves a good deal of sysadmin work, mostly Windows networks for small business customers. I ran into the above error on a Dell Server 2016 machine when trying to uninstall Windows components (Hyper-V in this case). This post gave me hint I needed to figure out the root cause, some missing language packs.

Now the original post recommends reinstalling the OS, which is a huge non-starter for me in an environment with a single file/AD server. The long fix starts with uninstalling language packs using the lpksetup tool and then manually removing references to any missing packs under the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\PackageDetectregistry subkeys. There are lots of them, literally thousands.

Just one of the 7600 registry values that need to be filtered.

I really needed to resolve this, so I spent an hour writing a PowerShell script to run through each subkey value and remove the one’s that referenced a missing language pack. In my case it was a German edition, so we’re searching for the string ‘~de-DE~’ below: 

$string = "*~de-DE~*"

$keys = Get-ChildItem -Path "hklm:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\PackageDetect"

foreach ($key in $keys) {
    (Get-ItemProperty -Path $key.PSPath).psobject.properties |where-object{$_.name -like $string} | Remove-ItemProperty -path $key.PSPath

}

It’s pretty simple, but was frustrating because of the need to get the .psobject.properties. I went through a lot of iterative testing to make sure that I targeted the proper values. Hopefully this helps someone else avoid a reinstallation. After running this script I was able to remove the Windows Hyper-V feature with no problems. I assume that this error was caused by an aborted uninstallation of one of the language packs. I’m not sure how they got on there, but assume that it was a Dell image load or something.

Anyways, cheers!

Posted on

Overwhelmed discovery

I’ve settled into a bit of a rhythm lately, spending my days working on Python code from home, continuing my project to integrate the various vendor that I use as part of my job via their APIs. In the evenings, after the girls have gone to bed, I’ve been working on various personal and school projects, or watching various computer science related videos. Friday to Saturday evenings, however, we’ve been doing a tech Shabbat, so I’ve been trying to find ways to get the girls out of the house as much as possible. More on that another time.

I finished watching the MIT Lisp series a week or two ago. It’s hard to believe that it’s a freshman-level class, as it’s fair to say that it’s probably more challenging than anything I’ve taken at my current university other than autotonoma theory. It covers so much, and Lisp, or Scheme, rather, has such a simple syntax, that it is really interesting to see how everything can be built up from these simple data structures and procedures.

Since I finished those, I’ve been watching a lot of Pycon videos to try and level up my understanding of some of the more advanced programming idioms that Python has. I’ve finally wrapped my head around list and dictionary comprehension, and have finally figured out what generators are good for. I’m going to to be exploring async and threading next.

In the last post, I was talking about the Ameritrade API and the work I’ve been doing to wrap my head around that. Well, yesterday I logged into my account and saw a notice that they were being bought out by Charles Schwab. Since I’ve been rather frustrated with the whole debugging process for that, I’ve decided to step away from that for a few days and decide whether it makes sense to continue with that or not. I don’t get the sense hat Schwab has a a very open system, and I’m not really enthusiastic about starting from scratch. Maybe I can find a broker with an open, standards compliant API that I can rollover to?

One of the main vendors that we use at work has a SOAP API that uses XML to transmit data. I spent several days trying to figure out how all the WSDL definitions work to see if I could use some metaprogramming to dynamically inspect the various entity attributes. Creating XML queries via chained method calls seems overly complicated, so I’ve been looking at how Django builds their SQL queries using QuerySet calls like Entity.objects.get(id='foo'). It’s much simpler, but it’s such a higher-level design program that I’ve become overwhelmed.

In general, that’s been the feeling lately, overwhelmed. It’s a different type of feeling than I have when I’m overcome with personal obligations and scheduling items. Now, it’s more like my programming ideas are getting to the point where it’s getting hard to intellectually manage them. The solution at this point seems to be to step back and work on something else, and give my brain time to work on the problem offline. An idea might pop into my head on it’s own, or a programming video might give me an idea on how to tackle a problem.

I’ve been trying to stay focused, by limiting the number of things that I’m allowing myself to work on, whether software projects or learning a new piano piece. But sometime you have to step back, and apply skills in to other problems that may reveal solutions in unexpected ways.

Posted on

QuickType and Ameritrade’s API.

My life goal of automating my job out of existence continues unabated. I’ve been spending a lot of time dealing with the APIs of the various vendors that we deal with, and I’ve spent a lot of time pouring over JSON responses. Most of these are multi-level structures, and usually leads to some clunky accessor code like object['element']['element']. I much rather prefer the more elegant dot notation of object.element.element instead, but getting from JSON to objects hasn’t been something I’ve wanted to spend much time on. Sure, there are a few options to do this using standard Python, but QuickType is by far the best solution out there.

I’ve been using the web-based version for the past few days to create an object library for Ameritrade’s API. Now first off, I’m probably going overboard and violating YAGNI (you ain’t gonna need it) principles by trying to include everthing that the API can return, but it’s been a good excuse to learn more about JSON schemas.

JSON schema with resultant Python code on right.

One of the things that I wish I’d caught earlier is that the recommended workflow in Quicktype is to start with example JSON data, and convert it to a JSON schema before going from that schema to your target language. I’d been trying to go straight from JSON to Python, and there were some problems. First off, the Ameritrade schema has a lot more types than I’ll need: there are two subclasses of securities account, and 5 different ones for the various instrument class. I only need a small subset of that, but thankfully Quicktype automatically combines these together. Secondly, Ameritrade’s response summary, both the schema and the JSON examples, aren’t grouped together in a way that can be parsed efficiently. I spent countless hours trying to combine things into a schema that is properly referenced and would compile properly.

But boy, once it did. Quicktype does a great job of generating code that can process JSON into a Python object. There are handlers for all of the various data types, and Quicktype will actually type check everything from ints to lists, dicts to unions (for handling Nones), and will process classes back out to JSON as well. Subobject parsing works very well. And even if you don’t do Python, it has a an impressive number of languages that it outputs to.

One problem stemming from my decision to use Ameritrade’s response summary JSON code instead of their schema is that the example code uses 0 instead of 0.0 where a float would be applicable. This led to Quicktype generating it’s own schema using integers instead of the JSON schema float equivalent, number. Additionally, Ameritrade doesn’t designate any properties as required, whereas Quicktype assumes everything in your example JSON is, which has led to a lot of failed tests.

Next, I’ll likely figure out how to run Quicktype locally via CLI and figure out some sort of build process to use to keep my object code in sync with my schema definitions. There’s been a lot of copypasta going on the past few days, and having it auto update and run tests when the schema changes seems like a good pipeline opportunity. I’ve also got to spend some more time understanding how to tie together complex schema. Ameritrade’s documentation isn’t up to standard, so figuring out to break them up into separate JSON objects and reference them efficiently will be crucial if I’m going to finish converting the endpoints that I need for my project.

That said, Quicktype is a phenomenal tool, and one that I am probably going to use for other projects that interface with REST APIs.

Posted on

Jacobin: War Is a Racket

It seems wholly appropriate to be covering this issue on Veteran’s Day. Both my parents were Army, and I’ve been living in an area of the country with one of the largest populations of active-duty personnel in the country.

This issue came to my door looking like a mock up of an old GI-Joe action toy, the packaging made out with images of our hero in the midst of battle. In this case, however, the included figurine is long after the battle has ended. Our action hero is sporting non-regulation long hair and beard, as well as a prosthetic leg, cane, and several bottles of prescription medications litter his feet.

This issue pulls no punches, deflating the notion of ‘service’ and ‘supporting our troops’. There’s plenty in this hefty issue about ending American imperialism, but probably the standout for me is the re-framing of American military culture as a ‘poverty draft’:

“The military welfare state only makes an effective recruiting tool because the Unite States denies all of us the civilian safety net we deserve. The US working class is held hostage by a political and military elite that exploits our deprivation to fuel its endless wars, forcing workers to make a devil’s bargain in pursuit of basic protections that should be available for all.

This statement hit me with such a moment of realization that reading it, I was almost embarassed that I had not seen it before. It’s a bit difficult to state the way which military culture permeates the culture here, so it was a bit like the David Foster Wallace bit about a fish learning what water is for the first time.

There’s a bit about activist opposition to ROTC programs in High Schools that made me think about the recruiting emails and texts that I’ve been getting through my college email address. And there’s a lot more in this issue, which is heftier than most of the others I’ve seen from Jacobin. They have a breakdown of the current 2020 Democratic Presidential contenders, (dl;dr: Biden, F; Warren, D-; Bernie: A-), infographic timelines on US military installations post-WWII, and some other features that are interesting.

But the short end of it is that they’re right about the hold that US militarism has on culture. From ‘Defense’ spending, displays of patriotism at sporting events, to the exploitation of Veterans by for-profit colleges via the GI Bill, American’s have an unhealthy relationship with our Armed Forces. And while a good deal of this issue does talk about concrete steps that can be taken to turn the tide, it seems like it might take generations before we have a population willing to fight back against our military-industrial system. Providing Medicare for All and free college would do a lot to break this, but then again, this may be exactly why the powers-that-be are fighting so hard to stop it.