QuickType and Ameritrade’s API.

My life goal of automating my job out of existence continues unabated. I’ve been spending a lot of time dealing with the APIs of the various vendors that we deal with, and I’ve spent a lot of time pouring over JSON responses. Most of these are multi-level structures, and usually leads to some clunky accessor code like object['element']['element']. I much rather prefer the more elegant dot notation of object.element.element instead, but getting from JSON to objects hasn’t been something I’ve wanted to spend much time on. Sure, there are a few options to do this using standard Python, but QuickType is by far the best solution out there.

I’ve been using the web-based version for the past few days to create an object library for Ameritrade’s API. Now first off, I’m probably going overboard and violating YAGNI (you ain’t gonna need it) principles by trying to include everthing that the API can return, but it’s been a good excuse to learn more about JSON schemas.

JSON schema with resultant Python code on right.

One of the things that I wish I’d caught earlier is that the recommended workflow in Quicktype is to start with example JSON data, and convert it to a JSON schema before going from that schema to your target language. I’d been trying to go straight from JSON to Python, and there were some problems. First off, the Ameritrade schema has a lot more types than I’ll need: there are two subclasses of securities account, and 5 different ones for the various instrument class. I only need a small subset of that, but thankfully Quicktype automatically combines these together. Secondly, Ameritrade’s response summary, both the schema and the JSON examples, aren’t grouped together in a way that can be parsed efficiently. I spent countless hours trying to combine things into a schema that is properly referenced and would compile properly.

But boy, once it did. Quicktype does a great job of generating code that can process JSON into a Python object. There are handlers for all of the various data types, and Quicktype will actually type check everything from ints to lists, dicts to unions (for handling Nones), and will process classes back out to JSON as well. Subobject parsing works very well. And even if you don’t do Python, it has a an impressive number of languages that it outputs to.

One problem stemming from my decision to use Ameritrade’s response summary JSON code instead of their schema is that the example code uses 0 instead of 0.0 where a float would be applicable. This led to Quicktype generating it’s own schema using integers instead of the JSON schema float equivalent, number. Additionally, Ameritrade doesn’t designate any properties as required, whereas Quicktype assumes everything in your example JSON is, which has led to a lot of failed tests.

Next, I’ll likely figure out how to run Quicktype locally via CLI and figure out some sort of build process to use to keep my object code in sync with my schema definitions. There’s been a lot of copypasta going on the past few days, and having it auto update and run tests when the schema changes seems like a good pipeline opportunity. I’ve also got to spend some more time understanding how to tie together complex schema. Ameritrade’s documentation isn’t up to standard, so figuring out to break them up into separate JSON objects and reference them efficiently will be crucial if I’m going to finish converting the endpoints that I need for my project.

That said, Quicktype is a phenomenal tool, and one that I am probably going to use for other projects that interface with REST APIs.

API obsession

I have been obsessed with APIs lately. Obsessed. Part of this stems from the interest in coding, of course, but part of it has come from a new focus on automating a lot of manual processes out of existence. I think I first really started messing around with them via crypto — of course — through the need to maintain price tracking sheets for my spec mining projects. I wanted to be able to keep track of the amount of coins that we were mining, the current price of said assets, and use that to calculate earnings and so forth. When I started tracking, I would manually get the prices from the exchange, paste them into a Google Drive doc, then copy my totals from one tab into a running monthly sheet. It quickly became tiresome, and when I found an add-on that someone had created to do lookups via CoinMarketCap (CMC), I became very interested in figuring out how it was done.

Eventually, I got interested in projects that weren’t available via this CMC interface, and had to start rolling my own. I was able to write Google scripts that could call the APIs of various exchanges and mining pools, to give me exchange totals, prices, and mining payouts. I’ve added them to a hodge-podge collections of scripts that I maintain in a sheet, so I can keep track of the entire venture. I use them to plan trades and track positions afterward. Of course Google Sheets has its limitations, and most of my work is in Python, but the basic premise is the same. Wrap an API request in a function wrapper, do something interesting with the result.

A lot of the interest also comes from my interest in automation. I’ve read the stories about people who have automated their jobs using Python, for example, and one of the fun things about APIs is that not only can you get information out of them, but you can send requests to them and make them do things for you. To stick with fintech for a bit longer, trade execution platforms are a perfect example of this. Being able to send orders to a trading platform through an API has enabled the high-frequency trading and bots to take over the markets. But my main interest is a bit closer to home, or work, to be more precise.

At my day job, we use several different systems to maintain our operation. The crux of it is a professional services automation (PSA) ticketing system and a remote monitoring and management (RMM) system. The two vendors that we use are integrated petty well. There are several major players in the space, and most of them plug together pretty well. The main issue is that the PSA requires a lot of manual setup and steps to do basic things like setting up new clients, configuring contracts, maintaining inventory. All which require multiple steps through their rather clunky UI. It’s a pain. Even something as simple as closing a ticket requires 4-5 mouse clicks.

Using the PSA’s API, I’ve begun to draft a collection of function that will allow me to close a ticket using a simple close_ticket(ticketID) call. I’ve developed more complicated functions that will create contracts, add products to those contracts and link assets from the RMM to those contracts. Right now I’m focused on standardizing operations across our clients, but there’s further opportunity to standardize operations between all of our franchise partners.

But perhaps the most critical opportunity that I’m focusing on within my day job is eliminating failures caused by human error.