When you ask an LLM to write code, it is effectively drawing from it's knowledge of every piece of code that it has ever seen and then probabilistically trying to produce the result that you are looking for.

LLMs don't "think" and by this I mean that they don't create new thoughts. Everything has to have existed in some capacity for the LLM to then work it's magic and then recite back to you in your context to solve your problem.

It works astonishingly well, but it has limitations.

In my experience the three places where it struggles the most are:

  • "Write this novel code for me to solve a problem that no one has solved."
  • "Reason over this large codebase that is so big that even the relevant parts are too large to fit into the context window."
  • "Start a new project completely from scratch to build this thing."

We need AGI to truly get new thought. Context windows will get big enough over the next year to mitigate the second case. We can solve the last point though, we just need to provide the LLM with a Starter Kit.

What is the actual problem when starting a new project?

You might not even know this is an issue. Bolt/Vercel/Replit do some version of a starter kit for you automatically. However I find the results of prompts to widely vary without establishing a base.

Typically it needs to include:

  1. Project setup with initial dependencies depending on your framework(s)
  2. A Component library for your UI.
  3. Enough of a user experience to login to the application. This isn't the full UX and should be an existing component library configured to get just the basic elements on the page.
  4. An Authentication system that allows someone to login. Most of these have automatic setup or generators. Make that the starting point.
  5. Unit and Functional testing with tests to cover the above basics.

The idea is that when the LLM starts working on tasks there should be enough in place that it has some rails to follow.

I think of it like this: most webpages or web apps these days are some important content inside of a frame that is the context of the application. You want the LLM to be able to focus on the content inside the frame, not the frame itself.

A lot of this is covered by generators. Back in the days when I used to work with Ruby on Rails this is something that Rails excelled at. There was a generator to solve just about every setup problem and gems to add functionality outside of the application logic.

In order to keep the context of your LLM focussed on your application and business logic you should run these generators ahead of time. Work within conventions. Leverage the automation. That is your starter kit.

Let's look at an example. I'm picking Elixir for my sample starter kit because there are many existing kits for something like Next.js and React.

An Elixir Starter Kit

Here's an inventory of what we think we need in place to get this working:

  1. A new Elixir project, based on the Phoenix Framework with the latest version of LiveView installed.
  2. Configuration for the project to used UUIDs (binary ids) as primary keys in the database. (This includes changes to tell Postgres to autogenerate the UUIDs.)
  3. A Schema folder, so that all Schema files are stored in a dedicated schema directory. (Keeping the schema separate makes it really easy to share just your schema with an LLM.)
  4. Phoenix Auth generators already run to create authentication and tests.
  5. Custom code needed to get Passkey support in place (it's custom because everyone does it wrong. :) Never store a binary DER in your database. Use the text-based PEM format!)
  6. SaladUI (the LiveView port of shadcn/ui).
  7. Enough UX to allow someone to login and logout of the Application. This is basically customization of the Root and Application layout and removal of the Phoenix boilerplate.
  8. AWS/S3 configured for file uploads (or pick your poison).
  9. Tests, formatters, linters and checkers for all of the above.
  10. Github actions to run the tests, checkers, etc.