Overwhelmed discovery

I’ve settled into a bit of a rhythm lately, spending my days working on Python code from home, continuing my project to integrate the various vendor that I use as part of my job via their APIs. In the evenings, after the girls have gone to bed, I’ve been working on various personal and school projects, or watching various computer science related videos. Friday to Saturday evenings, however, we’ve been doing a tech Shabbat, so I’ve been trying to find ways to get the girls out of the house as much as possible. More on that another time.

I finished watching the MIT Lisp series a week or two ago. It’s hard to believe that it’s a freshman-level class, as it’s fair to say that it’s probably more challenging than anything I’ve taken at my current university other than autotonoma theory. It covers so much, and Lisp, or Scheme, rather, has such a simple syntax, that it is really interesting to see how everything can be built up from these simple data structures and procedures.

Since I finished those, I’ve been watching a lot of Pycon videos to try and level up my understanding of some of the more advanced programming idioms that Python has. I’ve finally wrapped my head around list and dictionary comprehension, and have finally figured out what generators are good for. I’m going to to be exploring async and threading next.

In the last post, I was talking about the Ameritrade API and the work I’ve been doing to wrap my head around that. Well, yesterday I logged into my account and saw a notice that they were being bought out by Charles Schwab. Since I’ve been rather frustrated with the whole debugging process for that, I’ve decided to step away from that for a few days and decide whether it makes sense to continue with that or not. I don’t get the sense hat Schwab has a a very open system, and I’m not really enthusiastic about starting from scratch. Maybe I can find a broker with an open, standards compliant API that I can rollover to?

One of the main vendors that we use at work has a SOAP API that uses XML to transmit data. I spent several days trying to figure out how all the WSDL definitions work to see if I could use some metaprogramming to dynamically inspect the various entity attributes. Creating XML queries via chained method calls seems overly complicated, so I’ve been looking at how Django builds their SQL queries using QuerySet calls like Entity.objects.get(id='foo'). It’s much simpler, but it’s such a higher-level design program that I’ve become overwhelmed.

In general, that’s been the feeling lately, overwhelmed. It’s a different type of feeling than I have when I’m overcome with personal obligations and scheduling items. Now, it’s more like my programming ideas are getting to the point where it’s getting hard to intellectually manage them. The solution at this point seems to be to step back and work on something else, and give my brain time to work on the problem offline. An idea might pop into my head on it’s own, or a programming video might give me an idea on how to tackle a problem.

I’ve been trying to stay focused, by limiting the number of things that I’m allowing myself to work on, whether software projects or learning a new piano piece. But sometime you have to step back, and apply skills in to other problems that may reveal solutions in unexpected ways.

QuickType and Ameritrade’s API.

My life goal of automating my job out of existence continues unabated. I’ve been spending a lot of time dealing with the APIs of the various vendors that we deal with, and I’ve spent a lot of time pouring over JSON responses. Most of these are multi-level structures, and usually leads to some clunky accessor code like object['element']['element']. I much rather prefer the more elegant dot notation of object.element.element instead, but getting from JSON to objects hasn’t been something I’ve wanted to spend much time on. Sure, there are a few options to do this using standard Python, but QuickType is by far the best solution out there.

I’ve been using the web-based version for the past few days to create an object library for Ameritrade’s API. Now first off, I’m probably going overboard and violating YAGNI (you ain’t gonna need it) principles by trying to include everthing that the API can return, but it’s been a good excuse to learn more about JSON schemas.

JSON schema with resultant Python code on right.

One of the things that I wish I’d caught earlier is that the recommended workflow in Quicktype is to start with example JSON data, and convert it to a JSON schema before going from that schema to your target language. I’d been trying to go straight from JSON to Python, and there were some problems. First off, the Ameritrade schema has a lot more types than I’ll need: there are two subclasses of securities account, and 5 different ones for the various instrument class. I only need a small subset of that, but thankfully Quicktype automatically combines these together. Secondly, Ameritrade’s response summary, both the schema and the JSON examples, aren’t grouped together in a way that can be parsed efficiently. I spent countless hours trying to combine things into a schema that is properly referenced and would compile properly.

But boy, once it did. Quicktype does a great job of generating code that can process JSON into a Python object. There are handlers for all of the various data types, and Quicktype will actually type check everything from ints to lists, dicts to unions (for handling Nones), and will process classes back out to JSON as well. Subobject parsing works very well. And even if you don’t do Python, it has a an impressive number of languages that it outputs to.

One problem stemming from my decision to use Ameritrade’s response summary JSON code instead of their schema is that the example code uses 0 instead of 0.0 where a float would be applicable. This led to Quicktype generating it’s own schema using integers instead of the JSON schema float equivalent, number. Additionally, Ameritrade doesn’t designate any properties as required, whereas Quicktype assumes everything in your example JSON is, which has led to a lot of failed tests.

Next, I’ll likely figure out how to run Quicktype locally via CLI and figure out some sort of build process to use to keep my object code in sync with my schema definitions. There’s been a lot of copypasta going on the past few days, and having it auto update and run tests when the schema changes seems like a good pipeline opportunity. I’ve also got to spend some more time understanding how to tie together complex schema. Ameritrade’s documentation isn’t up to standard, so figuring out to break them up into separate JSON objects and reference them efficiently will be crucial if I’m going to finish converting the endpoints that I need for my project.

That said, Quicktype is a phenomenal tool, and one that I am probably going to use for other projects that interface with REST APIs.

Jacobin: War Is a Racket

It seems wholly appropriate to be covering this issue on Veteran’s Day. Both my parents were Army, and I’ve been living in an area of the country with one of the largest populations of active-duty personnel in the country.

This issue came to my door looking like a mock up of an old GI-Joe action toy, the packaging made out with images of our hero in the midst of battle. In this case, however, the included figurine is long after the battle has ended. Our action hero is sporting non-regulation long hair and beard, as well as a prosthetic leg, cane, and several bottles of prescription medications litter his feet.

This issue pulls no punches, deflating the notion of ‘service’ and ‘supporting our troops’. There’s plenty in this hefty issue about ending American imperialism, but probably the standout for me is the re-framing of American military culture as a ‘poverty draft’:

“The military welfare state only makes an effective recruiting tool because the Unite States denies all of us the civilian safety net we deserve. The US working class is held hostage by a political and military elite that exploits our deprivation to fuel its endless wars, forcing workers to make a devil’s bargain in pursuit of basic protections that should be available for all.

This statement hit me with such a moment of realization that reading it, I was almost embarassed that I had not seen it before. It’s a bit difficult to state the way which military culture permeates the culture here, so it was a bit like the David Foster Wallace bit about a fish learning what water is for the first time.

There’s a bit about activist opposition to ROTC programs in High Schools that made me think about the recruiting emails and texts that I’ve been getting through my college email address. And there’s a lot more in this issue, which is heftier than most of the others I’ve seen from Jacobin. They have a breakdown of the current 2020 Democratic Presidential contenders, (dl;dr: Biden, F; Warren, D-; Bernie: A-), infographic timelines on US military installations post-WWII, and some other features that are interesting.

But the short end of it is that they’re right about the hold that US militarism has on culture. From ‘Defense’ spending, displays of patriotism at sporting events, to the exploitation of Veterans by for-profit colleges via the GI Bill, American’s have an unhealthy relationship with our Armed Forces. And while a good deal of this issue does talk about concrete steps that can be taken to turn the tide, it seems like it might take generations before we have a population willing to fight back against our military-industrial system. Providing Medicare for All and free college would do a lot to break this, but then again, this may be exactly why the powers-that-be are fighting so hard to stop it.

Wired: September 2019

Three Years of Misery Inside Silicon Valley’s Happiest Company, by Nitasha Tiku:The cover story of this month’s issue a really in-depth piece about the chaos that has been plaguing of the eponymous internet search company, Google. One of the things I love about Wired is their long form reporting, and this article is several thousand words, about 14 pages of text in nine parts. Tiku details the leaks from inside the company that seemingly destroyed the company’s unspoken rule of ‘what happens at Google, stays in Google’. Social justice activists tore down the company’s missteps with regard to sexual harrassment by managers and execs, betraying their “don’t do evil” motto, through dealings with authoritarian China and the United States military.

The article really opens up a look at the inner culture of Google, how they had a fairly open culture, with C-level executives making themselves open for questioning from staffers, and with rank-and-file employees creating sub-cultures within the company. T

The story starts in January 2017, as employees took to the streets after the Trump administration’s travel ban was declared. The company stood behind their employees and stood up for immigrant rights. Then in June, an engineer named James Damore release a 10-page memo ‘explaining’ why there weren’t more female engineers in the industry. Damore was reacting to efforts to promote female engineers within the company, and claimed that this was a bad idea since there were biological reasons why there aren’t more women in STEM fields.

The eventual backlash to Damore’s memo, and his eventual dismissal, started a culture war between the company’s conservatives. Apparently, this minority within Google had been existing within their own corner of the company, but following this, some of them became emboldened and began to step up their opposition and trollishness. They doxed several of the liberal organizers, thereby breaking the company’s sacred rule of non-disclosure.

This episode is just one of several that Tiku details. By the end of the piece, it’s clear that Google’s culture has been transformed, and that while their employees may still be sticking to their ‘don’t be evil’ motto, the executives of the company, driven by shareholder capitalist growth demands, have lost their way.

FAN-tastic Planet: When I was a teenager, growing up in the mid 90’s, Wired was the coolest magazine on the planet. I felt that it offered up both a vision of the future and secret knowledge about where things were headed. Wired was an essential fuel for the ideas that eventually led to my career in computers and programming. Now, having learned more about the nascent cyberpunk culture that Wired killed off in favor off Dot Com boom and bust, that I wonder more about what could have been. I bring this up because I was almost shocked to read the intro to this special culture section in this issue.

“A person sitting at a computer – it was a mystical sight. Once,” it opens, before going further into something straight out of a Rushkoff monologue: “The users, we humans were the almight creator-gods of the dawning digial age, and the computers did our bidding. We were in charge. In today’s world, subject and object have switched places… Computers run the show now, and we -mere data subjects.” It’s almost like they literally took Rushkoff’s point about ‘figure and ground’ verbatim. Please forgive me, dear reader, for mistaking if his point isn’t original. But given his disdain for Wired’s entire raison d’etre during the 90’s and aughts, I find it entirely ironic that they have this section: fan-fic-writing nerds; Netflix’s turn toward Choose-Your-Own-Adventure-style programming; social-media influencers ‘connecting’ with fans over special, subscriber only feeds; and the rise of a new generation of crossword-puzzle writers who are bringing new, diverse voices to another field traditionally dominated by white men. (This last one actually includes a crossword, and let to several days of puzzling on my part.)

Free Riders, by Zeynep Tufekci: The front pages of this issue have the standard Wired fare, gadgets and the latest tech. This time it’s smart-writing tools and augmented-reality gizmos. A bit about the current NASA Mars rover being built, another extolling the joys of world-building video games, another bemoaning Facebook’s creepy dating feature. I wanted to end with a mention of Tufekci’s bit about the prevalence of commercial companies that are built on top of the free, open source tools that have been released to the internet. Not that this is a problem, per se, but there have been many instances of packages that have been so instrumental to the success of these companies, and to the industry in general, that have had security issues, or depended on the unpaid efforts of some very overworked contributors. Tufekci details two pertinent examples: the Heartbleed bug, which affected the OpenSSL spec used to protect almost all web traffic, and core-js, a Javascript library widely used in web browsers.

In the latter case, the developer had been working on the library almost every day for five years without pay, and had solicited less than a hundred dollars. While some might blame him for not taking advantage of his project’s popularity and using it to leverage himself into a high-paying gig somewhere, the issue highlights a problem with the web’s altruistic origins, that have long since been abused by corporation. At least, in the case of OpenSSL, they were able to guilt some firms into providing more funding, but we’ve got a long way to figure out a way to reward these open source programmers that have provided the tools that the web is built on.

The Nation: Aug 26/Sept 2, 2019

We’re seriously behind, both on the pile of periodicals that we have to read and on the ones that we’ve read that we haven’t covered. Letting them age a bit this way puts the coverage in perspective a bit, and helps justify our procrastination.

News You Can Lose, by John Nichols: I have yet to watch any of the Democratic Presidential debates. I gave up my binge-drinking, live Tweeting, event watching after Trump was elected president. Douglass Rushkoff has noted how the debate as television spectacle gave us our current, reality show president, and I am in no mood to participate in the current round. That’s not to say that I haven’t post-watched some of the more ‘gotcha’ moments of the current crop: Biden’s numerous flubs, Julian Castro’s miscalculated attack on Biden, and of course, Elizabeth Warren’s onstage murder of one of the other also-rans.

Nichols column follows the Democratic debate in Detroit, hosted by Fox News. There was a union solidarity event a few hours before the debate at a General Motors transmission plant that was scheduled to be closed the day after the debate. Nichols notes that Dems would have been smart to have hosted the debate at the union hall, or at a church across the street from the actual debate’s location, where Detroit’s Democratic congressional delegation were attending in solidarity with families facing deportation. Either of these would have been smart choices to help focus the debates on substantive policy issues.

But of course, that isn’t the point. Spectacle is. Nichols’s point is that progressives need to make more of an effort to wrest control back of these debates back from the party and the networks and points to the People’s Presidential Forum as an example of this. The Forum, which was to be hosted in October by New England group Rights and Democracy, was cancelled because “not enough candidates could make this date work”.

The American Workplace, by Bryce Covert: Workplace discrimination against pregnant women is rampant in America, especially against working-class women. My wife was able to save up weeks of leave for both of our two children, but for most American women, finding or keeping a job while pregnant can be difficult, and employers use a variety of measures to screen out or dismiss these women. Pay discrimination, or more specifically, the gender-based wage gap, and the Equal Rights Amendment are important political issues today, and Covert profiles the challenges of several women whose lives have been affected by this issue and are fighting back.

Without repeating the details of these cases, I should note that America is one of the only industrialized countries that does not provide for paid maternal care for new mothers. And this fact has many secondary effects on the well-being of the child, including potential educational and economic ones. The profit-driven war on expectant mothers is a roadblock to economic mobility. We should grant American mothers the same privileges as the rest of the developed world, and allow them the time to bond with their newborns and not force them to either give up this time with their newborns, or give up on their careers.

Source: Pew Research Center

Vigilant Struggle, by Robert Greene II: Review of Stony the Road, by Henry Louis Gates. The new Watchmen HBO series premieres with scenes from what can only be described as a race riot: Tulsa, Oklahoma, 1921. A black couple with a young boy tries to escape violence in the city as white citizens indiscriminately shoot unarmed blacks while buildings burn around them. Later, explosive devices are dropped from airplanes onto a garage where others had been hiding out. The scene was so outlandish that I thought it was some sort of alternate history being built around the show’s background. It wasn’t until later that I discovered that the racially motivated destruction depicted in the show was based on actual events.

My own ignorance of the Tulsa race riot almost a hundred years ago is further magnified by the history of Reconstruction following the Civil War. Henry Louis Gates has produced a new documentary series for PBS titled Reconstruction, and his book Stony The Road is a companion to this series. According to Greene, Gates has attempted to expand the period defined by Reconstruction to encompass the war itself as well as the first couple decades of the nineteenth century. Whether he includes the 1921 Tulsa riot in this definition remains to be seen.

Greene, a history professor at South Carolina’s Calfin University, spends several thousand words on the subject of Reconstruction before getting into Gates’s documentary, and notes how there was really two Reconstructions: one that was reconciliatory toward the vanquished South, an another, more radical Reconstruction that attempted to redefine the entire concept of American democracy and expand it to the former enslaved peoples. He notes that the period was one of revanchanist backlash, and lynchings, and raises questions about just how successful these latter efforts at reform were.

Gates documentary, he notes, provides a level of context to the African American experience, and is successful at detailing the evidence of continued aggressions against the freed slaves: racist stereotypes in papers and books, minstrel shows, the founding of the Klan, Jim Crow. This evidence is held as proof against claims of a modern post-racial America. Ultimately the Reconstruction is “not just about the rise and fall of black power in post-Civil War America, but the the rise and fall of black equality in all spheres of American life, cultural, political and otherwise.”

Confirmation bias

So I have proved once again that drinking sucks. I went over one hundred days, probably closer to 110, and have spent the past several days back in old habits. Nothing bad has happened, but nothing good has happened either. I’ve actually fallen off of several habits, dear reader, so I am hoping that coming back here and writing will help drive the demons out. I’m being melodramatic, so I guess I should explain.

I had settled into a bit of a routine, waking up early, meditating, drinking some tea, fasting, turning off screens at 10PM and going to bed at a decent hour. I felt like I was getting a lot done, and I felt great. Then I guess I settled into a few bad habits that started a decline. I’ve been drinking way to many caffeinated drinks, and then started staying up too late. I justified it cause I’ve been watching the MIT CS videos. But I wasn’t getting good sleep, so more caffeine, and so on and so on.

Before I stopped drinking, I had bought a bottle of wine, and after I ceased someone gave me a bottle of Scotch. The two bottles were sitting next to each other on a hutch in the dining room, next to the other drinking paraphernalia. I could see them every day, and as long as they were there they functioned as sort of a totem. I knew they were there, and I was proving something to myself by not choosing to drink them every day. And hopefully, days would pass that the thought would never even cross my mind.

Of course, I do not live alone. And I’ve never tried to impose my abstinence on my spouse. I may have even picked up something for her at the store a time or two. But on a particular day this last week, she had a ‘really bad day’ and wanted something to drink, but the only thing that was left was the wine and the scotch. After a bit of half-hearted protestration, I opened the wine. And I poured two glasses. Cause I would be damned if I was going to let her drink my bottle of wine without me. And so it was began.

The next day, so the seal on the scotch was broken. And over the next few nights, finger after finger I drank. And so on and so on, until I was buying and drinking an entire six pack of IPA until well after midnight this morning. I wasn’t hung over, but as I laid in bed this morning I determined that I would get back to the schedule. Wake. Meditate. Write. Be present.

This is my confession. And this is my reset.