Navigating the Ups and Downs of AI and Blockchain Development: A Real-World Saga

A futuristic workspace with advanced AI and blockchain technology, featuring screens displaying complex code and digital interfaces, amidst a modern, sleek office environment. The setting is vibrant yet professional, embodying the cutting-edge nature of AI and blockchain development.

From Technical Hurdles to Unexpected Downtimes – Lessons Learned in the Trenches of Tech Innovation

In the ever-evolving world of AI and blockchain, we often encounter a myriad of challenges, some anticipated and others completely unforeseen. Our recent experience serves as a testament to this unpredictable journey. Last Friday, we started encountering issues with our OpenAI assistant, deployed through a Chainlit app. For those unfamiliar, Chainlit is akin to Streamlit but with a Python backend and a React frontend, a blend of technological sophistication.

Our initial task seemed straightforward – uploading a PDF and several markdown files as source documents to enhance our app’s user experience. However, the challenge lay in properly displaying these citations in the UX. The complexities of this task led us down a rabbit hole of technical intricacies. We faced peculiar issues with the model’s response, especially when it came to retrieving information from the database. An instance that stood out was when a query about taking a day off on a birthday yielded no relevant results from the documents, leading to confusing model responses.

Determined to resolve this, we dived deeper, fine-tuning our approach. We realized that when no results were found, it was best to assume the documents didn’t cover the queried topic. Yet, this was just the tip of the iceberg. As we delved further, testing and tweaking, more bizarre responses emerged. The model reported technical difficulties in reading documents, yet contradictorily, it displayed search results.

Amidst this chaos, we discovered another glaring issue – the redundancy of file uploads. We had inadvertently uploaded the same file numerous times, a clear oversight in our workflow. This necessitated a thorough cleanup and recreation of our assistants, ensuring everything functioned seamlessly in our playground tests and in the app.

The real twist came when we recognized that the root of our troubles lay not within our code or approach but with OpenAI’s retrieval runs. Their models were not performing as expected, leading us to some prompt engineering adjustments. Frustrated and exhausted, we decided to pause and revisit the problem with a fresh perspective on Monday.

Monday brought its own set of surprises. Our return was greeted by a non-functional app, thanks to Chainlit Cloud being down. This downtime was a significant blow as Chainlit Cloud is integral to our data persistence layer, storing user maps and conversation databases. It’s akin to how ChatGPT displays threads of conversations. This persistence is crucial for our app’s functionality.

Rushing to the Chainlit Discord server, which I hadn’t joined until then, I learned that their cloud service was undergoing an update. This revelation was both horrifying and enlightening. In response, we disabled data persistence on our production server as a temporary fix and spent the day refactoring our code to adapt to the new Chainlit Cloud version.

This refactoring journey was not just about code. It was about understanding and integrating changes from the Chainlit cookbook, separating our custom code, and preserving the essence of third-party contributions. We faced dilemmas about merging different code histories and ensuring our customizations, especially around the assistant’s citation returns, were seamlessly integrated.

Our approach was to move our customizations into a separate file and import Chainlit’s updated cookbook as a utility. This method, while effective, brought its own set of challenges. Chainlit functions uniquely, with Python on the backend and a React frontend, a system that allows for deployment on platforms like Replit. Our goal was to customize this frontend without bloating the repository.

The downtime also highlighted a critical migration issue with Chainlit Cloud, now rebranded as Literal AI. They had changed key determinations and OAuth methods, leading to a temporary loss of our historical data. This situation underlined the importance of staying attuned to dependencies and the risks involved in relying on external services.

Despite these challenges, we emerged with valuable insights and a stronger, more resilient application. Our journey with Chainlit and OpenAI continues, marked by both triumphs and tribulations. As we progress, we remain committed to exploring and harnessing the immense potential of AI and blockchain technology, ready to tackle whatever hurdles come our way.

Navigating the Evolving Landscape of LLM Apps and Agent Frameworks

Exploring the Convergence of Workflow Management and AI in the Age of Automation

In the dynamic and rapidly evolving field of LLM applications and agent frameworks, staying ahead of the curve is a necessity. This blog post delves into our journey and insights as we navigate through various platforms and technologies, aiming to integrate advanced AI capabilities into our workflow and products.

Steamship to Super Agent: A Journey of Exploration

Our exploration began with the Steamship framework, offering a comprehensive environment for single-agent application development. However, as our needs evolved, we found ourselves seeking more control over the systems we run. This led us to Super Agent, a self-hosted solution offering a similar range of capabilities but with greater autonomy.

Integrating OpenAI’s Latest Innovations

A significant focus has been on integrating OpenAI’s latest offerings. We’re currently working on incorporating the Whisper model into Discord voice channels, aiming to enable the model to transcribe and respond using text-to-speech modules. This integration represents a step towards a fully multimodal business intelligence system.

The Promise of ChainLit and TaskWeaver

ChainLit has emerged as a compelling option for building Python LLM applications, akin to StreamLit but tailored for LLM apps. Simultaneously, we’re examining Microsoft’s TaskWeaver, a code-first agent framework that appears to merge workflow management with LLM capabilities seamlessly.

Navigating the Challenges of Observability and Management

Observability remains a significant challenge, especially given the limitations of OpenAI’s organizational structure. Tools like Pezzo Labs and Langfuse offer promising solutions, but the decision on which platform to commit to remains open.

Looking Forward: An Eye on the Future

Our journey is characterized by continuous learning and adaptation. We recognize the importance of not committing prematurely to any single framework, keeping our options open as the landscape of LLM apps and agent frameworks continues to evolve.

References and Further Reading: