From Technical Hurdles to Unexpected Downtimes – Lessons Learned in the Trenches of Tech Innovation
In the ever-evolving world of AI and blockchain, we often encounter a myriad of challenges, some anticipated and others completely unforeseen. Our recent experience serves as a testament to this unpredictable journey. Last Friday, we started encountering issues with our OpenAI assistant, deployed through a Chainlit app. For those unfamiliar, Chainlit is akin to Streamlit but with a Python backend and a React frontend, a blend of technological sophistication.
Our initial task seemed straightforward – uploading a PDF and several markdown files as source documents to enhance our app’s user experience. However, the challenge lay in properly displaying these citations in the UX. The complexities of this task led us down a rabbit hole of technical intricacies. We faced peculiar issues with the model’s response, especially when it came to retrieving information from the database. An instance that stood out was when a query about taking a day off on a birthday yielded no relevant results from the documents, leading to confusing model responses.
Determined to resolve this, we dived deeper, fine-tuning our approach. We realized that when no results were found, it was best to assume the documents didn’t cover the queried topic. Yet, this was just the tip of the iceberg. As we delved further, testing and tweaking, more bizarre responses emerged. The model reported technical difficulties in reading documents, yet contradictorily, it displayed search results.
Amidst this chaos, we discovered another glaring issue – the redundancy of file uploads. We had inadvertently uploaded the same file numerous times, a clear oversight in our workflow. This necessitated a thorough cleanup and recreation of our assistants, ensuring everything functioned seamlessly in our playground tests and in the app.
The real twist came when we recognized that the root of our troubles lay not within our code or approach but with OpenAI’s retrieval runs. Their models were not performing as expected, leading us to some prompt engineering adjustments. Frustrated and exhausted, we decided to pause and revisit the problem with a fresh perspective on Monday.
Monday brought its own set of surprises. Our return was greeted by a non-functional app, thanks to Chainlit Cloud being down. This downtime was a significant blow as Chainlit Cloud is integral to our data persistence layer, storing user maps and conversation databases. It’s akin to how ChatGPT displays threads of conversations. This persistence is crucial for our app’s functionality.
Rushing to the Chainlit Discord server, which I hadn’t joined until then, I learned that their cloud service was undergoing an update. This revelation was both horrifying and enlightening. In response, we disabled data persistence on our production server as a temporary fix and spent the day refactoring our code to adapt to the new Chainlit Cloud version.
This refactoring journey was not just about code. It was about understanding and integrating changes from the Chainlit cookbook, separating our custom code, and preserving the essence of third-party contributions. We faced dilemmas about merging different code histories and ensuring our customizations, especially around the assistant’s citation returns, were seamlessly integrated.
Our approach was to move our customizations into a separate file and import Chainlit’s updated cookbook as a utility. This method, while effective, brought its own set of challenges. Chainlit functions uniquely, with Python on the backend and a React frontend, a system that allows for deployment on platforms like Replit. Our goal was to customize this frontend without bloating the repository.
The downtime also highlighted a critical migration issue with Chainlit Cloud, now rebranded as Literal AI. They had changed key determinations and OAuth methods, leading to a temporary loss of our historical data. This situation underlined the importance of staying attuned to dependencies and the risks involved in relying on external services.
Despite these challenges, we emerged with valuable insights and a stronger, more resilient application. Our journey with Chainlit and OpenAI continues, marked by both triumphs and tribulations. As we progress, we remain committed to exploring and harnessing the immense potential of AI and blockchain technology, ready to tackle whatever hurdles come our way.