This is a non-static page for describing how I am thinking about using LLMs.

Harness

When you start working with language models to generate code, you want whatever is produced to work. This requires some additional harnesses that both guide the agent and also give the agent a way to verify what it is doing. The tools that were helpful for humans are mostly useful for agents as well. They may require a bit of editing and changing of format to be more useful for an agent than a human.

Below are some general properties of a harness that are helpful.

Entrypoint to each session

Providing context is the key activity in prompting agents to generate code, but extensive and detailed specification does not have to be manually injected on each interaction. It is important to keep sessions scoped to a reasonable amount of change and providing too much context is something to be avoided. The important context can be fetched via skills or provided in important files that the agent will naturally interact with while performing its work. This way you don’t have to start a session with extensive copy pasting of specification right at the start, instead the specification can be set in files which then gives the agent opportunity to update all that info as it naturally changes over time.

Goal setting

A session should have a scoped goal. Additionally, a good goal is able to be expressed as a paragraph. If more information is needed at the start of the session, consider instead adjusting the harness to already have that required information. The better you equip the harness to solve the problem, the easier it should be for an agent to solve it. This also helps one to avoid throwing in a huge problem description directly into the agent before thinking about how you think the problem should be solved.