Text-to-SQL: Build vs buy

Why Text-to-SQL?

The concept of text-to-SQL is as old as SQL itself and has long been the dream of many data professionals. Simplifying the complex world of SQL into an easily understood, natural language-based interface could democratize data access and drive insights across organizations.

However, achieving this goal hasn't been easy. The complexity of the problem is often underestimated. What appears to be a simple request like "Show me top sales by customers" unravels a litany of sub-questions - how do we define a customer, what account table should we use, how are "sales" defined?

Only recently, with the emergence of generative AI through Large Language Models (LLMs), has a viable solution been possible. But even then, it's not as simple as feeding a user's question and database schema to the LLM and expecting accurate results. The journey from query to results is a complex pipeline, with numerous steps involved to craft the correct prompts for the LLM.

So, should you build this solution in-house or should you buy?

Building: A Custom but Resource-Intensive Solution‍

Embarking on building a text-to-SQL solution in-house can offer significant benefits:

1. Tailored to Your Needs: Building in-house can address specific, complex requirements that off-the-shelf solutions might not cater to, especially if you have a complex data storage or partitioning system.
2. In-Depth Understanding: Building a solution in-house can lead to a deeper understanding of the system, enhancing your organization’s ability to optimize and innovate.

However, there is a real cost to building:

1. Technical Demands: Building a text-to-SQL solution requires a multi-disciplinary team—data engineers, back-end engineers, AI/prompt engineers, and front-end engineers. This might strain resources, especially for smaller organizations. Expect the team to dive deep into solutions like OpenAI, Anthropic, LlamaIndex or Pinecone. Building an accurate prompt pipeline requires many complex integrations.
2. Maintenance Burden: Custom solutions demand ongoing commitment. Regular monitoring, model tuning, and adapting to new LLM releases are necessary to ensure optimal performance. As we’ve recently seen with GPT4, models can regress and it’s important to have the agility to switch models (and hint: each model responds to the same prompt very differently).

‍Buying: Fast Implementation but Less Bespoke

The alternative is purchasing a pre-built solution, which can offer its own advantages:

1. Quick Deployment: Purchased solutions can be deployed rapidly, delivering value and improving productivity without the time and resource investments needed for building an in-house system.
2. Vendor Support: The vendor will take responsibility for system maintenance, including updates for new LLM releases and accuracy adjustments.
3. Deeper Integrations: Your pre-build self-service BI solution for ad hoc questions should connect to all your systems, whether it be your data warehouse, your semantic layer or your BI. Building and maintaining all these integrations can be challenging.

However, the "buy" route is not without pitfalls:

1. Compatibility Concerns: While many pre-built solutions aim for broad compatibility, there may still be instances where integration with your existing systems can be challenging. You may use a lesser known or homegrown semantic layer that isn’t widely supported for example.
2. Customization Limits: Especially at the data layer, if your data model is extremely complex and normalized with subtleties that may be lost even with the best LLM, building your own system may be better. However, at this point the best option may be to even start with addressing data tech debt and building the right foundations.

Build if you’re able to go very deep technically, buy if you’re aiming for cost and value efficiency

While building a text-to-SQL solution using generative AI offers the potential for a highly customized system, it comes with substantial resource requirements and ongoing maintenance obligations. In contrast, buying a pre-built solution promises swift deployment and vendor support but may cause challenges if your data stack is already bespoke or leverages niche solutions.

Ultimately, the decision hinges on your organization's specific needs, resources, and long-term goals. As you navigate this critical juncture, it's essential to carefully weigh the merits and challenges of both paths and align your choice with your strategic objectives. Whether buying or building, the end goal remains the same: harnessing the power of LLMs to democratize data access and drive informed decision-making across your organization.