Pennykoin is dead! Long live Pennykoin!
About two weeks ago, Jerry Howell, the main developer of Pennykoin (Pk), a Cryptonote privacy coin, publicly abandoned the project, citing personal health and financial reasons. As someone who was heavily involved with Pk during its early days, (eighteen months ago!), and probably the only person besides Jerry that has looked under the hood of Pk, I’ve been asked to take up maintenance of the project.
I feel it is necessary to provide some details that should be shared with the Pk community, my assessment of the current state of the project, and why I think Jerry was right to abandon it.
I don’t have much to say about Pk’s origins. Jerry posted an [ANN] link on May 14, 2018, which was the day after he had the release version of the wallet software released. There isn’t really much to say about Pk as a product. Cryptonote is a privacy coin framework. It was developed to allow people to create their own Proof of Work (PoW) coin by manipulating things such as block time and emissions rate. Bytecoin was the first coin made from it, (and is probably a scam), and Monero is based off of it as well.
Pk first came to my attention over the summer through a Tweet of a noted shitcoiner that I followed, and there seemed to be a nice community developing for Pk on Twitter, so I threw some hashrate at it and started mining. Jerry and I started talking, and pretty soon I was heavily involved, setting up the official mining pool and block explorer for Jerry while he worked on the code.
I found Jerry’s backstory very interesting. He worked in the service industry, I want to say it was food-related, but my memory is hazy. He had been sidelined due to health issues and was off his feet, so he started learning to program as a way to pass the time. I was actually impressed with what he had managed to accomplish. He had a vision for Pk, and I was happy to be helping build something, instead of passively investing in a project as I had been up until then.
Signs of trouble
At some point during the summer, it became clear to me that there were some serious issues that were going to hamper Pk in the long run.
The first was a problem with bootstrapping, which is the way in which a freshly copy of the PK software downloaded the historical blockchain data from the other nodes. In short, it didn’t work. Part of the point of a blockchain is the validation process, in which a node verifies that each block is valid and meets the parameters of the chain that have been specified. Now Cryptonote is built to allow these parameters to be changed. So if one initially specifies a mining reward of 10, and later decides to change this to be 6, there is a way to specify versions of the blockchain, that the node can use to validate.
Unfortunately, Jerry made several changes to the Pk code in the first few weeks that didn’t follow this pattern, and as a result, the nodes were unable to download the complete chaindata, and would fail to sync when connecting to the network. Jerry’s response to this was to provide the chaindata as a separate download that needed to be thrown into the application data folder on machines.
I attempted to help fix this, but my knowledge of CN and C++ programming wasn’t up to the task, so I tried to hack our way around it by disabling validation checks on certain blocks. I don’t believe that these workarounds made it into any surviving releases of the Pk source, but the bootstrap issue has only gotten worse and worse, which is now why new installs require a 100+ megabyte download to bootstrap clients to block 130,000 or so, which is around the time of Jerry’s last update around the end of August.
Chain fail #1
Pk started to get noticed around late summer, and we started to see more and more hashpower being added to the network. I forget exactly what algorithm Jerry was using for PoW, but it was GPU-friendly, and available on rental services like NiceHash. Pk wasn’t yet available on any exchanges yet, but we had a healthy over-the-counter (OTC) market going on. (Disclaimer: I was providing escrow services for a fee.) Now, I can’t really say for certain whether what happened next was malicious or not, but we experienced a chain failure to to what I refer to as a difficulty attack.
Without getting too technical, block chains have a target block time, the average time in which a block should be mined. In the case of bitcoin, it’s two minutes. In Pk, it was two. There is an adjustment built into the blockchain algorithms that will determine whether this target block time is being met, and it will adjust the PoW difficulty target accordingly. To explain this in plain language, think of the PoW game as a requirement that n coins must be flipped heads in a row to win the block reward. As more miners join the network, that number will increase, from 10 to 100 to 1000 and so on. As miners leave the network, that number decreases.
What happened with Pk that fall, was that so much hash power was thrown at the network that some individual or individuals were able to mine an immense amount of Pk at a faster than normal rate. This increased the target difficulty immensely, at which point this power was removed from the network. The end result was that the remaining Pk miners did not have enough power available to mine any blocks, or at least at any rate that would have kept the chain running. Miners, not getting the expected rewards, left the network, making the promise even worse.
Now, I give Jerry immense credit for getting things running again. He basically implemented a difficulty adjustment to the code base and swapped the mining algorithm out for one which we could deploy much more powerful ASIC miners. He pushed the changes out after a week, the community threw some hashpower at it, and we were off and running again.
Note that I am being a bit speculative about what happened, without running metrics on block times and other chaindata, one cannot know whether this was a simple difficulty attack or something more nefarious.
You don’t know what you don’t know
In spite of the heroic efforts on the part of Jerry and others to keep Pk moving after this, there were problems with his development style that were due to his novice, that have ultimately hobbled this project. Now, let me be clear, I have nothing but respect for Jerry and am saying none of this to impinge his character. That said, I think there are things that any potential Pk investor needs to know, and that’s why I want to put this on the public record. I am by no means a professional software developer, so while I was ultimately able to spot these problems, I was in no position to correct them myself.
First off, Jerry never caught the hang of version control. When I first started working with him, he had a horrible habit of committing to master, and when he needed to change something, or hit a wall, he would just delete everything and start from scratch, which breaks the commit change and makes tracking changes very difficulty. For example, here’s what the early releases of the PKNode software looks like:
Now, someone with more experience than I could probably rebase these to link them together, but there are dozens and dozens of commits and branches that go no where, and since I stepped away from Pk at the end of fall 2018, I don’t really know what he was doing. There are a few blog updates over the past year that talks about various initiatives, but there’s just too many changes for me to try and reverse engineer.
Technical debt must be paid
Unconfirmed transactions or ill-gotten gains
There are a number of problems with Pk as it stands now. First off, there
is was a huge bug that has locked up about seven percent of all of the current coins in circulation. For a quick technical explanation, I need to talk about the unlock_time parameter of a CN transaction. This unlock time is primarily meant as a way to prevent newly mined coins from being spent immediately. I believe it’s meant to discourage forking attacks or something similar. With Pk, this unlock time is normally set to 10 blocks (20 minutes) for block rewards. For standard transactions, this is zero. Normally, anyways.
After Jerry’s disappearance, following the latest release, we started getting reports from people with unconfirmed transactions. According to the information that I’ve gathered, these people withdrew these amounts from the Graviex exchange, and their wallets were not making them available to be sent elsewhere. I was able to get my hands on one of the individual wallets, and confirm the issue.
A number of these transactions were mined to the blockchain with a large unlock time. So large, in fact, that at Pk’s target block time, it will be one hundred and twenty years before these funds are available to the owners. Now I wrote a program to examine the entire Pk blockchain, and I believe that these transactions were all limited to between November 2 and March 31, which coincides with the release of several version of the Pk software. I do not believe this bug is currently in circulation. However, I have been unable, or more accurately, unwilling, to contact Graviex to see what was going on their end in order to properly establish a root cause. In fact, I don’t think it’s likely that I will.
There are other issues with the current code base that I’m not go into much detail on. There seems to be discrepancies between the CLI and GUI software, which I believe stems from the core CN/Pk libraries existing separately. Wallets amounts are different between the two for some of my older wallets, and I’ve documented at least one transaction that is present in the CLI is not displayed at all in the GUI. From what I can determine, this is not related to deposit functionality, which is not implemented in the CLI at all.
Given the recent vulnerabilities and bugs that have recently been exposed in Cryptonote, I cannot rule out the possibility that this was intentional, whether it was malicious or more benign. I am not accusing anyone specific of any ill behavior, but am just stating that I do not know enough to make that determination at this time.
The Jerry-sized hole
For the most part, up until now I’ve been discussing what are mostly technical challenges that can be overcome with enough time and effort. There is one issue that cannot, and that is the absence of Jerry Howell.
I’m passing no judgement here. Jerry had to chose between health and keeping a roof over his head, and I’m not going to second-guess his decision. Unfortunately, he couldn’t eke out a living from Pk, and priorities is priorities. He did what he had to do. For the future of Pk, that presents us with some problems.
First off, because Jerry still holds the key to the kingdom, so to speak, we would have to recreate much of the public-facing infrastructure: two sets of github repos, social media accounts, various Discord instances, &c.. Thankfully, members of the governance committee have access to the web domains and hosting services that are running the main pool. But in order to modify the code, I would have to create new Pk repos, which would present us with a different type of ‘fork’ that could become problematic. The last thing we want is a non-authoritative code source online, and what would happen if we took steps to publish new code to maintain Pk, and Jerry changes his mind and decides to go in a different direction?
To that end, members of the governance committee have decided that it would be best to wait till the end of the month to see if he will resurface. There is no doubt that he has been active; we noticed a new Github repo for an unrelated project that he created a few days after he disappeared. The best thing for Pk is that he will pop up long enough to properly hand off control of these resources so that development can continue without him.
If not, it will likely be the end of ‘Pennykoin’ altogether. The risk of associating with that name is too great, and at a minimum, the community will have to decide on a new name and rebrand.
The future of Pk
Even if Jerry does come back and hands things off, it doesn’t mean that we’ll be out of the woods. I’ve mentioned the technical debt that has to be paid off, and I can tell you right now that I am not the one to lead Pk out of the desert, so to speak. My personal investment in this project is not substantial, and the opportunity cost to me to spend the necessary amount of time needed for me to figure out how to fix what has happened is too great a risk. Unfortunately, there does not seem to be another individual within the community right now that has the technical skills needed to manage a project of this complexity. I simply do not have the time.
A quick calculation of the current Pk supply will give us a market cap of Pennykoin: 1.6b Pk minted over 150k blocks. The ‘price’ as I write this is less than one sat, more like 4/10 of one, which means we’re looking at a total cap of less than 7BTC, or $70,000 in USD. That’s impressive nonetheless, and demonstrates the promise of crypto-based assets, but as Jerry has found, one can’t pay the rent with those kinds of numbers. I just can’t commit to that, and am not willing to take responsibility as the one to fix the numerous problems that exist today. I’m not even sure I know how to fix these problems.
My personal and professional opinion is that the entire codebase should be redeployed from the ground up using test-driven development principles, with features selected from a proper governance process. I also think that inevitably, the existing chain will need to be abandoned completely. I’m not familiar enough with blockchain engineering that I could figure out a way to hard fork the stuck funds back into circulation. It would be something on par to Ethereum undoing the DAO hack, and I’ll admit I’m not up for it. have already been thinking about protocols for an automated chain swap that would allow us to provide current Pk holders with equity in the new chain.
I realize that this post may not be what anyone wants to hear, but I think it is an honest assessment of where Pk stands. I haven’t given much thought to what the fallout of this will be, but I assume that most people will be disappointed, and maybe even angry at me for publishing this. So be it. I’ve been in the cryptospace for almost five years, and I think this is one of the greatest opportunities that our generation will see in our lifetimes. Many of you no doubt agree, else you would not be here at the end of this brutal, brutal bear market. That said, I believe that there is no such thing as failure so long as knowledge is gained. I know I have learned an immense amount about blockchains from working on this, and many others have done so, and stepped up to run nodes, websites, or other parts of the Pk infrastructure.
So there is a way for us all to move forward, utilizing the connections that we have. It may not look like Pk does today, but there’s no reason that the connections and network that we’ve built should go to waste. This space is moving fast, and there are lots of opportunities to form a new type of organization that can help us move forward and build a future that we can all be proud of. I am personally very interested in work being done around Decentralized Autonomous Organizations (DAO) and Decentralized Finance (DeFi), but others may have other suggestions that they may be interested in.
What’s important to me is that the future that the Pk community decides on, whatever the course, is decided in a democratic manner. That is our first challenge, ensuring that what we build is more robust, and can’t be destroyed by the decision of one person to disappear. I have already been in discussions with others on the governance committee about forming a formal, legal entity to help manage this responsibility.
I realize the community is going to need some time to process what I’ve written, and I know there will be lots of discussion around this in the days to come. I hope that enough people will feel it worth their time to stick around, and help decide on next steps, together. There is vast opportunity in the crypto space, and I have already met many people that I would be honored to work with to build the next iteration of blockchain based products.