Becoming a Git-xpert

I have been trying to get a grip on the Pennykoin CLI code base for some time. One of the problems that I’ve had is that the original developer had a lot of false starts and stops, and there’s a lot of orphan branches like this:

Taken with GitKraken

If that wasn’t bad enough, at some point they decided to push the current code to a new repo, and lost the entire starting commit history. Whether this was intentional or not, I can’t say. It’s made it very tricky for me to backtrack through the history of the code and figure out where bugs were introduced. So problem number one that I’m dealing with is how to link these two repos together so that I have a complete history to search through.

Merging two branches

So we had two repos, which we’ll call pk_old and pk_new. I originally tried methods where I tried to merge the repos together using branches, but I either wound up with the old repo as the last commit, or with the new repo and none of the old history. I spent a lot of time going over my bash history file and playing with using my local directories as remote sources, deleting and starting over. Then I was able to find out that there was indeed a common commit between these two repos, and that all I had to do was add the old remote with the –tags option to pull in everything.

mkdir pk_redux
cd pk_refresh
git init
git remote add -f pk_new https://github.com/Pennykoin/Pennykoin-old.git --tags
git merge pk_new/master
git remote add -f pk_old https://github.com/Pennykoin/Pennykoin-old.git --tags

Now, I probably could have gotten away by just cloning the pk_new repo instead of initializing an empty directory and adding the remote, but we the end result should be the same. A quick check of the tags between the two original repos and my new one showed that everything was there.

The link between the two repos

Phantom branches

One of the things that we have to do as part of our pk_redux, as we’re calling it, is setup new repos that we actually have control over. This time around, everything will be setup properly as part of governance, so that I’m not the only one with keys to the kingdom in case I go missing. I want to take advantage of GitLab’s integrated CI/CD, as we’ve talked about before, so I setup a new group and pkcli repo. I pushed the code base up, and saw all the tags, but none of the branches were there.

The issue ultimately comes down to the fact that git branches are just pointers to a specific commit in a repository’s history. Git will pull the commits down from a remote as part of a fetch job, but not the pointers to those branches unless I physically checked them out. Only after I created these tracking branches on my local repo could I then push them to the new remote origin.

Fixing Pennykoin

So now that I’ve got a handle on this repo, my next step is to hunt some bugs. I’ll probably have to do some more work to try and de-orphan some of these early commits in the repo history, cause that will be instrumental in tracking down changes to the Cryptonote parameters. These changes are likely the cause for the boostrap issue that exists. And my other priority is figuring out if we can unlock the bugged coins. From there I’d like to implement a test suite, and make sure that there is are proper branching workflows for code changes.

The Rise and Fall of Pennykoin

Pennykoin is dead! Long live Pennykoin!

About two weeks ago, Jerry Howell, the main developer of Pennykoin (Pk), a Cryptonote privacy coin, publicly abandoned the project, citing personal health and financial reasons. As someone who was heavily involved with Pk during its early days, (eighteen months ago!), and probably the only person besides Jerry that has looked under the hood of Pk, I’ve been asked to take up maintenance of the project.

I feel it is necessary to provide some details that should be shared with the Pk community, my assessment of the current state of the project, and why I think Jerry was right to abandon it.

Beginnings

I don’t have much to say about Pk’s origins. Jerry posted an [ANN] link on May 14, 2018, which was the day after he had the release version of the wallet software released. There isn’t really much to say about Pk as a product. Cryptonote is a privacy coin framework. It was developed to allow people to create their own Proof of Work (PoW) coin by manipulating things such as block time and emissions rate. Bytecoin was the first coin made from it, (and is probably a scam), and Monero is based off of it as well.

Pk first came to my attention over the summer through a Tweet of a noted shitcoiner that I followed, and there seemed to be a nice community developing for Pk on Twitter, so I threw some hashrate at it and started mining. Jerry and I started talking, and pretty soon I was heavily involved, setting up the official mining pool and block explorer for Jerry while he worked on the code.

I found Jerry’s backstory very interesting. He worked in the service industry, I want to say it was food-related, but my memory is hazy. He had been sidelined due to health issues and was off his feet, so he started learning to program as a way to pass the time. I was actually impressed with what he had managed to accomplish. He had a vision for Pk, and I was happy to be helping build something, instead of passively investing in a project as I had been up until then.

Signs of trouble

At some point during the summer, it became clear to me that there were some serious issues that were going to hamper Pk in the long run.

Bootstrap fail

The first was a problem with bootstrapping, which is the way in which a freshly copy of the PK software downloaded the historical blockchain data from the other nodes. In short, it didn’t work. Part of the point of a blockchain is the validation process, in which a node verifies that each block is valid and meets the parameters of the chain that have been specified. Now Cryptonote is built to allow these parameters to be changed. So if one initially specifies a mining reward of 10, and later decides to change this to be 6, there is a way to specify versions of the blockchain, that the node can use to validate.

Unfortunately, Jerry made several changes to the Pk code in the first few weeks that didn’t follow this pattern, and as a result, the nodes were unable to download the complete chaindata, and would fail to sync when connecting to the network. Jerry’s response to this was to provide the chaindata as a separate download that needed to be thrown into the application data folder on machines.

I attempted to help fix this, but my knowledge of CN and C++ programming wasn’t up to the task, so I tried to hack our way around it by disabling validation checks on certain blocks. I don’t believe that these workarounds made it into any surviving releases of the Pk source, but the bootstrap issue has only gotten worse and worse, which is now why new installs require a 100+ megabyte download to bootstrap clients to block 130,000 or so, which is around the time of Jerry’s last update around the end of August.

Chain fail #1

Pk started to get noticed around late summer, and we started to see more and more hashpower being added to the network. I forget exactly what algorithm Jerry was using for PoW, but it was GPU-friendly, and available on rental services like NiceHash. Pk wasn’t yet available on any exchanges yet, but we had a healthy over-the-counter (OTC) market going on. (Disclaimer: I was providing escrow services for a fee.) Now, I can’t really say for certain whether what happened next was malicious or not, but we experienced a chain failure to to what I refer to as a difficulty attack.

Without getting too technical, block chains have a target block time, the average time in which a block should be mined. In the case of bitcoin, it’s two minutes. In Pk, it was two. There is an adjustment built into the blockchain algorithms that will determine whether this target block time is being met, and it will adjust the PoW difficulty target accordingly. To explain this in plain language, think of the PoW game as a requirement that n coins must be flipped heads in a row to win the block reward. As more miners join the network, that number will increase, from 10 to 100 to 1000 and so on. As miners leave the network, that number decreases.

What happened with Pk that fall, was that so much hash power was thrown at the network that some individual or individuals were able to mine an immense amount of Pk at a faster than normal rate. This increased the target difficulty immensely, at which point this power was removed from the network. The end result was that the remaining Pk miners did not have enough power available to mine any blocks, or at least at any rate that would have kept the chain running. Miners, not getting the expected rewards, left the network, making the promise even worse.

Now, I give Jerry immense credit for getting things running again. He basically implemented a difficulty adjustment to the code base and swapped the mining algorithm out for one which we could deploy much more powerful ASIC miners. He pushed the changes out after a week, the community threw some hashpower at it, and we were off and running again.

Note that I am being a bit speculative about what happened, without running metrics on block times and other chaindata, one cannot know whether this was a simple difficulty attack or something more nefarious.

You don’t know what you don’t know

In spite of the heroic efforts on the part of Jerry and others to keep Pk moving after this, there were problems with his development style that were due to his novice, that have ultimately hobbled this project. Now, let me be clear, I have nothing but respect for Jerry and am saying none of this to impinge his character. That said, I think there are things that any potential Pk investor needs to know, and that’s why I want to put this on the public record. I am by no means a professional software developer, so while I was ultimately able to spot these problems, I was in no position to correct them myself.

First off, Jerry never caught the hang of version control. When I first started working with him, he had a horrible habit of committing to master, and when he needed to change something, or hit a wall, he would just delete everything and start from scratch, which breaks the commit change and makes tracking changes very difficulty. For example, here’s what the early releases of the PKNode software looks like:

Orphan commits are not good.

Now, someone with more experience than I could probably rebase these to link them together, but there are dozens and dozens of commits and branches that go no where, and since I stepped away from Pk at the end of fall 2018, I don’t really know what he was doing. There are a few blog updates over the past year that talks about various initiatives, but there’s just too many changes for me to try and reverse engineer.

Technical debt must be paid

Unconfirmed transactions or ill-gotten gains

There are a number of problems with Pk as it stands now. First off, there is was a huge bug that has locked up about seven percent of all of the current coins in circulation. For a quick technical explanation, I need to talk about the unlock_time parameter of a CN transaction. This unlock time is primarily meant as a way to prevent newly mined coins from being spent immediately. I believe it’s meant to discourage forking attacks or something similar. With Pk, this unlock time is normally set to 10 blocks (20 minutes) for block rewards. For standard transactions, this is zero. Normally, anyways.

After Jerry’s disappearance, following the latest release, we started getting reports from people with unconfirmed transactions. According to the information that I’ve gathered, these people withdrew these amounts from the Graviex exchange, and their wallets were not making them available to be sent elsewhere. I was able to get my hands on one of the individual wallets, and confirm the issue.

A number of these transactions were mined to the blockchain with a large unlock time. So large, in fact, that at Pk’s target block time, it will be one hundred and twenty years before these funds are available to the owners. Now I wrote a program to examine the entire Pk blockchain, and I believe that these transactions were all limited to between November 2 and March 31, which coincides with the release of several version of the Pk software. I do not believe this bug is currently in circulation. However, I have been unable, or more accurately, unwilling, to contact Graviex to see what was going on their end in order to properly establish a root cause. In fact, I don’t think it’s likely that I will.

Et cetera…

There are other issues with the current code base that I’m not go into much detail on. There seems to be discrepancies between the CLI and GUI software, which I believe stems from the core CN/Pk libraries existing separately. Wallets amounts are different between the two for some of my older wallets, and I’ve documented at least one transaction that is present in the CLI is not displayed at all in the GUI. From what I can determine, this is not related to deposit functionality, which is not implemented in the CLI at all.

Given the recent vulnerabilities and bugs that have recently been exposed in Cryptonote, I cannot rule out the possibility that this was intentional, whether it was malicious or more benign. I am not accusing anyone specific of any ill behavior, but am just stating that I do not know enough to make that determination at this time.

The Jerry-sized hole

For the most part, up until now I’ve been discussing what are mostly technical challenges that can be overcome with enough time and effort. There is one issue that cannot, and that is the absence of Jerry Howell.

I’m passing no judgement here. Jerry had to chose between health and keeping a roof over his head, and I’m not going to second-guess his decision. Unfortunately, he couldn’t eke out a living from Pk, and priorities is priorities. He did what he had to do. For the future of Pk, that presents us with some problems.

First off, because Jerry still holds the key to the kingdom, so to speak, we would have to recreate much of the public-facing infrastructure: two sets of github repos, social media accounts, various Discord instances, &c.. Thankfully, members of the governance committee have access to the web domains and hosting services that are running the main pool. But in order to modify the code, I would have to create new Pk repos, which would present us with a different type of ‘fork’ that could become problematic. The last thing we want is a non-authoritative code source online, and what would happen if we took steps to publish new code to maintain Pk, and Jerry changes his mind and decides to go in a different direction?

To that end, members of the governance committee have decided that it would be best to wait till the end of the month to see if he will resurface. There is no doubt that he has been active; we noticed a new Github repo for an unrelated project that he created a few days after he disappeared. The best thing for Pk is that he will pop up long enough to properly hand off control of these resources so that development can continue without him.

If not, it will likely be the end of ‘Pennykoin’ altogether. The risk of associating with that name is too great, and at a minimum, the community will have to decide on a new name and rebrand.

The future of Pk

Even if Jerry does come back and hands things off, it doesn’t mean that we’ll be out of the woods. I’ve mentioned the technical debt that has to be paid off, and I can tell you right now that I am not the one to lead Pk out of the desert, so to speak. My personal investment in this project is not substantial, and the opportunity cost to me to spend the necessary amount of time needed for me to figure out how to fix what has happened is too great a risk. Unfortunately, there does not seem to be another individual within the community right now that has the technical skills needed to manage a project of this complexity. I simply do not have the time.

A quick calculation of the current Pk supply will give us a market cap of Pennykoin: 1.6b Pk minted over 150k blocks. The ‘price’ as I write this is less than one sat, more like 4/10 of one, which means we’re looking at a total cap of less than 7BTC, or $70,000 in USD. That’s impressive nonetheless, and demonstrates the promise of crypto-based assets, but as Jerry has found, one can’t pay the rent with those kinds of numbers. I just can’t commit to that, and am not willing to take responsibility as the one to fix the numerous problems that exist today. I’m not even sure I know how to fix these problems.

Stats on the PK block explorer

My personal and professional opinion is that the entire codebase should be redeployed from the ground up using test-driven development principles, with features selected from a proper governance process. I also think that inevitably, the existing chain will need to be abandoned completely. I’m not familiar enough with blockchain engineering that I could figure out a way to hard fork the stuck funds back into circulation. It would be something on par to Ethereum undoing the DAO hack, and I’ll admit I’m not up for it. have already been thinking about protocols for an automated chain swap that would allow us to provide current Pk holders with equity in the new chain.

Moving forward

I realize that this post may not be what anyone wants to hear, but I think it is an honest assessment of where Pk stands. I haven’t given much thought to what the fallout of this will be, but I assume that most people will be disappointed, and maybe even angry at me for publishing this. So be it. I’ve been in the cryptospace for almost five years, and I think this is one of the greatest opportunities that our generation will see in our lifetimes. Many of you no doubt agree, else you would not be here at the end of this brutal, brutal bear market. That said, I believe that there is no such thing as failure so long as knowledge is gained. I know I have learned an immense amount about blockchains from working on this, and many others have done so, and stepped up to run nodes, websites, or other parts of the Pk infrastructure.

So there is a way for us all to move forward, utilizing the connections that we have. It may not look like Pk does today, but there’s no reason that the connections and network that we’ve built should go to waste. This space is moving fast, and there are lots of opportunities to form a new type of organization that can help us move forward and build a future that we can all be proud of. I am personally very interested in work being done around Decentralized Autonomous Organizations (DAO) and Decentralized Finance (DeFi), but others may have other suggestions that they may be interested in.

What’s important to me is that the future that the Pk community decides on, whatever the course, is decided in a democratic manner. That is our first challenge, ensuring that what we build is more robust, and can’t be destroyed by the decision of one person to disappear. I have already been in discussions with others on the governance committee about forming a formal, legal entity to help manage this responsibility.

I realize the community is going to need some time to process what I’ve written, and I know there will be lots of discussion around this in the days to come. I hope that enough people will feel it worth their time to stick around, and help decide on next steps, together. There is vast opportunity in the crypto space, and I have already met many people that I would be honored to work with to build the next iteration of blockchain based products.

Regards,
el

How I became a (failed) blockchain engineer

Impostor syndrome is bit of a popular topic in certain professional circles, people being haunted by the nagging feeling that they don’t really know what they’re doing in their chosen field, and that soon someone will expose them as a fraud. It seems to be prevalent among software engineers and other fields where specialization inevitably informs one that the sum of what they know is but a fraction of the sum of knowledge. It’s not something that I content with, for the most part, but I bring it up because I’ve been thinking about when I get to call myself a software engineer.

Not that I’ve ever been lacking for confidence, rather as fair number of people of my gender and complexion, I suffer from an excess of it. I’ve always operated from a place where coming up to an answer to a question, or hypothesizing an answer as a means to discovering one is usually the correct course of action. This is different than having an opinion about something about which I know nothing, of course, but this invariably leads me to question or challenge assumptions to answers from others, some of who may have more expertise in a subject. As one who realizes that many advancements in various fields have been made by outsiders, I don’t see any problem with this. My wife on the other hand, calls it ‘male answer syndrome’ when I inevitably make a statement that runs counter to something that she knows as true, or has been led to believe by someone else with more expertise.

On Twitter, this has become known by the number of reply guys that plague women’s timelines. I don’t know if this is specific to the professional classes or not, but I assume it happens wherever a woman forms an opinion about anything at all. Around here though, I’ve started correcting my wife’s accusations of male answer syndrome with wrong answer syndrome, as our eldest daughter has picked up the annoying habit of declaring demonstrably false statements about things in the middle of an argument about some parenting slight against her.

I digress.

A little over a year ago, I became involved with a small blockchain project. I’ll keep it unnamed so as to prevent this from showing up in search engine results for now, but it was a fork of privacy coin framework that was being managed by a single developer. The dev was new to software design and had a lot of time on his hands due to disability, and had done a decent job of establishing a community around this project. I came in a few weeks after it had been launched, and became a part of the community. I set up mining pools, block explorers, and started doing a lot of commits to various side projects associated with this new cryptoasset.

Eventually though, I began to see warning signs. our dev didn’t understand git versioning very well, and had a bad habit of just starting new repos when he got stuck or a feature didn’t work out. The commit and branch graphs had a lot of dead ends and unconnected starts. I was never really sure what he was doing, what he was working on. I tried to help, fixing a couple things in C++ where I could and trying to help him get his workflow under some sort of change management, but there wasn’t a test suite, and several of the changes that he had introduced in the code base seemed to have killed backward compatibility of the existing blockchain.

In all, I think things became too overwhelming for our friend. I had to step out of the picture after a few months to deal with real life issues, and I don’t think anyone else ever stepped in that understood blockchain functionality the way that I had after spending hours looking into the code. In addition to maintaining the code base, trying to introduce new features, our friend had to deal with hash attacks which stalled out the blockchain completely not once, but twice, and had to deal with the constant support issues of pool operators, trading exchanges, and users with wallet or transaction issues.

It all became too overwhelming. In addition to dealing with all of this, our friend was also dealing with both physical and mental health issues, as well as money issues that apparently threatened his housing situation. One day, he published an apology to everyone for failing and went dark.

I hadn’t been active in the community as I had been, but still kept an eye or ear on things. The stalled chain had been restarted and a new version of the wallets released, and things seemed like they were getting back on the right track. There were a couple people that seemed to be having some issues with funds, but nothing serious. Most of the trouble seemed to be with burnout.

There was a panic, of course, among some of the miners and people who had been sticking with the project over the last few weeks of troubles. There were strong words about fraud and exit scams from people who had gotten involved with the project more recently. A few community leaders tried to calm nerves.

But ultimately, there was no one else that had any technical experience with the code itself. None that were willing to speak up, at least, and I don’t know of anyone else that had been interacting at the level that I had. And I still had a bit of financial and emotional stake in this project, which is why I volunteered to take the reins on a technical assessment, to figure out the state of the network and whether there was any chance of getting things back on track.

And that’s how I became responsible for a broken blockchain project. I’m still trying to figure out just how broken things are, whether it can be fixed, or whether a project that was worth hundreds of thousands of dollars is now dead.