feat(code-template): Add Code Template System and Tool Calling Infrastructure #302

thecodacus · 2024-11-16T05:55:16Z

Add Code Template System and Tool Calling Infrastructure

Overview

This PR introduces two major architectural enhancements:

A code template system for project scaffolding
A foundational infrastructure for AI tool calling with an agent-based architecture

These changes enable more sophisticated project initialization and lay the groundwork for AI-driven tooling capabilities.

Key Changes

1. Template System Implementation

Added project template import functionality
Implemented GitHub-based template fetching
Added template configuration and management
Introduced template ignore patterns for selective imports
Added support for template-specific prompts

2. Tool Calling Infrastructure

Implemented base agent architecture for tool execution
Added tool calling parser and execution pipeline
Integrated tool configuration management
Added support for agent-based routing and coordination
Implemented template variables system for dynamic prompting

3. Agent System

Added AgentPromptGenerator for structured prompt generation
Implemented AgentOutputParser for processing tool calls
Added coordinator agent for query routing
Implemented project scaffolding agent
Added support for agent registration and management

4. UI Enhancements

Added tool configuration toggle in chat interface
Implemented tool execution status display
Added template selection interface
Enhanced message parsing for tool responses
Added support for tool calling visualization

Technical Implementation

Agent Architecture

Base agent class with configurable tools and parameters
XML-based tool call protocol for structured communication
Support for template variables and dynamic prompt generation
Coordinator agent for intelligent query routing

Template System

GitHub API integration for template fetching
Support for .bolt configuration directory
Template ignore patterns for selective imports
Custom prompt support for templates

Tool Execution Pipeline

Tool call parsing and validation
Agent selection and routing
Parameter validation
Tool execution
Response formatting and streaming

Testing

Tested template import functionality
Verified tool calling pipeline
Tested agent routing system
Verified template ignore patterns
Tested tool configuration persistence

Migration Impact

No breaking changes for existing chat functionality
Tool calling requires explicit opt-in via toggle
Template system is additive to existing functionality
Existing message parsing remains unchanged

Future Enhancements

Add support for more complex tool chains
Enhance template management system
Add more specialized agents
Improve tool execution visualization
Add support for concurrent tool execution

Preview

Code.Template.Demo.mp4

thecodacus · 2024-11-16T06:48:16Z

@wonderwhy-er , sorry but I tried to make the changes as little as possible 😅

thecodacus · 2024-11-16T06:59:06Z

Also this approach is text based, to it can be used by any llm, as long as it follows the structured output and does not get confused. that require prompt tuning which can be done in separate task

we can now add agents for github project import or commit, using this framework

wonderwhy-er · 2024-11-16T10:03:23Z

Ok, will need to take a deeper look in to how it works

thecodacus · 2024-11-16T14:30:12Z

this is an example of git import using the tooling framework

Git.Import.using.Tool.Architecture.mp4

wonderwhy-er · 2024-11-16T18:55:02Z

Ahh its still big, will take time to fully review...

champ2050 · 2024-11-17T19:35:06Z

@thecodacus Awesome work! Thank you for your contribution. Have you had a chance to check out Microsoft's Magnetic-One? It includes an orchestrator that passes commands to specific agents, each designed to handle particular tasks. If you could take a look at it, that might make a fantastic addition to this project. Magnetic-One was recently released, so not many people have seen it yet, but it works really well. Thanks again for your awesome work, I really appreciate it!

thecodacus · 2024-11-18T15:32:40Z

no I didn't get the chance to see that yet. will definitely check that

Prathamesh-Shanbhag · 2024-11-22T15:29:13Z

This was an awesome addition - Now all I look forward to is the ability to have attachments as reference to the prompts and one click deploy to vercel or other similar platforms!!

wonderwhy-er · 2024-11-23T17:13:40Z

I did not even notice it had github integration in some form.
If I understand it correctly its for local use only, or one who hosts would need to add his key in there.

Considering its not oauth based all users of hosted app will have 5k requests per hour
https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28

We still need to get to oauth variant to work at some point.

thecodacus · 2024-11-23T21:04:26Z

I did not even notice it had github integration in some form. If I understand it correctly its for local use only, or one who hosts would need to add his key in there.

Considering its not oauth based all users of hosted app will have 5k requests per hour https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28

We still need to get to oauth variant to work at some point.

Yes, for code template system, template needs to be downloaded from github. we will need the key, but the api also has some allowance without any key as well. in future we can also modify the agent code to use the tarball download. that will not require any oauth or api key at all

for the github project import demo that i showed in the threaded video, was just to show case what we can do with this system, this PR does not contain the github agent included. that is something I just made with this system quickly to showcase

adriano2023666 · 2024-11-23T21:20:32Z

I am currently facing a NotFoundError in the application, and I need assistance in resolving it. The error message is as follows:

php
Copiar código
NotFoundError: Failed to execute 'insertBefore' on 'Node': The node before which the new node is to be inserted is not a child of this node.
at insertBefore (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b44fa53c:15324:26)
at insertOrAppendPlacementNode (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b44fa53c:24310:15)
at commitPlacement (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b44fa53c:24267:15)
at commitReconciliationEffects (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b44fa53c:24784:15)
...
The issue seems to be related to the insertBefore method in the DOM, where the new node cannot be inserted because the specified reference node is not a child of the parent node. This is causing the application to behave unexpectedly, and it's affecting the UI by preventing certain elements from being rendered or placed correctly.

Could anyone help me understand what might be causing this error and guide me on how to resolve

mrsimpson

Dear @thecodacus ,

First off: I am not a maintainer of this project nor have I fully understood all your decisions.
But I very much appreciate your contribution and wanted to show this be reviewing it.
Your PR is quite huge, so I am afraid it might take some time to get it merged. Therefore, I focused on "how could this be easier to merge"

Two things I noted which I personally would welcome in order to keep this project maintainable:

Along implementing your extensions, you added some improvements (such as adding proper types) which I appreciate a lot. This could be very well be one (or even multiple) separate small PRs.
I do think that modifying files from an upstream should be reduced as much as possible. Thus, I do think it would help to move blocks of code into separate small files and place them in dedicated folders for tool or agent.

In general, I do like your work a lot and hope my feedback helps to get this merged quicker.

Cheers, Oliver

app/components/chat/APIKeyManager.tsx

mrsimpson · 2024-11-24T11:11:01Z

app/components/chat/BaseChat.tsx

@@ -85,6 +86,7 @@ interface BaseChatProps {
  sendMessage?: (event: React.UIEvent, messageInput?: string) => void;
  handleInputChange?: (event: React.ChangeEvent<HTMLTextAreaElement>) => void;
  enhancePrompt?: () => void;
+


There are some formatting artifacts. Please lint:fix the source to keep the diff small as possible

mrsimpson · 2024-11-24T11:11:52Z

app/components/chat/BaseChat.tsx

@@ -113,8 +115,13 @@ export const BaseChat = React.forwardRef<HTMLDivElement, BaseChatProps>(
  ) => {
    const TEXTAREA_MAX_HEIGHT = chatStarted ? 400 : 200;
    const [apiKeys, setApiKeys] = useState<Record<string, string>>({});
+    const [toolConfig, setToolConfig] = useState<IToolsConfig>({


Thanks for making it an interface 👍

mrsimpson · 2024-11-24T11:13:17Z

app/components/chat/BaseChat.tsx

@@ -136,6 +143,25 @@ export const BaseChat = React.forwardRef<HTMLDivElement, BaseChatProps>(
      });
    }, []);

+    useEffect(() => {
+      const config = Cookies.get('bolt.toolsConfig'); 


WDYT about moving this whole piece of logic into a separate file in components/tools and to have only a single-line-modification here?

mrsimpson · 2024-11-24T11:14:20Z

app/components/chat/BaseChat.tsx

                />
-                {provider && (
+                <div className="flex justify-between items-center">


another candidate for a separate file

mrsimpson · 2024-11-24T12:05:53Z

app/lib/tools/base-tool.ts

+
+export abstract class BaseTool {
+    protected webcontainer: Promise<WebContainer>;
+    constructor(webcontainerPromise: Promise<WebContainer>) {


Shouldn't each tool have at least a name so it can identify itself in tool messages

Edit: Now I see that the agent has a name. I think I misunderstood what tool and agents are. Could you clarify this a bit?

mrsimpson · 2024-11-24T12:07:37Z

app/lib/tools/project-template-import.ts

you said this was not actually part of this PR, as far as I understood ;)

But since this is a viable implementation and doesn't add to the overall modifications, I appreciate it being in here!

mrsimpson · 2024-11-24T12:11:19Z

app/utils/constants.ts

@@ -117,6 +117,12 @@ const PROVIDER_LIST: ProviderInfo[] = [
  }
 ];

+const codeTemplates: TemplateInfo[] = [


Shouldn't this be part of either the agent or the toolConfig? I think it should outside this file at least

mrsimpson · 2024-11-24T12:11:52Z

app/utils/matchPatterns.ts

This is also actually independent from this PR, isn't it?

mrsimpson · 2024-11-24T12:13:01Z

app/utils/types.ts

Also here: Move it to app/lib/tools/types.ts . I doubt that a single types file is best practice.

including one review suggestion Co-authored-by: Oliver Jägle <[email protected]>

github-actions · 2024-12-06T03:00:43Z

This pull request has been marked as stale due to inactivity. If no further activity occurs, it will be closed in 7 days.

thecodacus added 3 commits November 16, 2024 04:49

feat(code-template): added code template and text based tool calling

8e9c528

chore: clean up

1b2662b

Merge branch 'main' into artifact-based-tool-calling

fce7a17

thecodacus changed the title ~~Artifact based tool calling~~ feat(code-template): Add Code Template System and Tool Calling Infrastructure Nov 16, 2024

thecodacus added 3 commits November 16, 2024 11:47

ui fix made room for tool toggle switch

d0c890c

chore: type fixes

d46070b

fix: message parser bugfix

d5d07e8

thecodacus marked this pull request as ready for review November 16, 2024 06:46

chore: reverted the log as it was in main

1bb6f99

thecodacus mentioned this pull request Nov 16, 2024

Project templates before start create #52

Closed

thecodacus added 3 commits November 16, 2024 17:19

fix: fixed sime bugs happened after resolving mer conflicts

7c5123a

Merge branch 'main' into artifact-based-tool-calling

be8598c

import project prompt tuning

98b605f

thecodacus mentioned this pull request Nov 20, 2024

feat(code-template): Addition of Code Template And Setting Up Foundation For AI Tool Calling Support #266

Closed

thecodacus added 2 commits November 24, 2024 13:44

fix: added skip reply on tool call when project reload

a456c5e

mostly prompt tuning and some failsafe

79c17d1

mrsimpson reviewed Nov 24, 2024

View reviewed changes

Apply suggestions from code review

0cf4e68

including one review suggestion Co-authored-by: Oliver Jägle <[email protected]>

github-actions bot added the stale The pull / issue is stale and will be closed soon label Dec 6, 2024

dustinwloring1988 removed the stale The pull / issue is stale and will be closed soon label Dec 8, 2024

mrsimpson mentioned this pull request Dec 10, 2024

feat(Filters): Add Middleware Filter Chain System to add user written customization to llm calls #614

Open

thecodacus mentioned this pull request Dec 21, 2024

website's initial framework templates missing #801

Open

thecodacus closed this Dec 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(code-template): Add Code Template System and Tool Calling Infrastructure #302

feat(code-template): Add Code Template System and Tool Calling Infrastructure #302

thecodacus commented Nov 16, 2024 •

edited

Loading

thecodacus commented Nov 16, 2024

thecodacus commented Nov 16, 2024

wonderwhy-er commented Nov 16, 2024

thecodacus commented Nov 16, 2024

wonderwhy-er commented Nov 16, 2024

champ2050 commented Nov 17, 2024

thecodacus commented Nov 18, 2024

Prathamesh-Shanbhag commented Nov 22, 2024

wonderwhy-er commented Nov 23, 2024

thecodacus commented Nov 23, 2024 •

edited

Loading

adriano2023666 commented Nov 23, 2024

mrsimpson left a comment

mrsimpson Nov 24, 2024

mrsimpson Nov 24, 2024

mrsimpson Nov 24, 2024

mrsimpson Nov 24, 2024

mrsimpson Nov 24, 2024

mrsimpson Nov 24, 2024

mrsimpson Nov 24, 2024

mrsimpson Nov 24, 2024

mrsimpson Nov 24, 2024

github-actions bot commented Dec 6, 2024

feat(code-template): Add Code Template System and Tool Calling Infrastructure #302

feat(code-template): Add Code Template System and Tool Calling Infrastructure #302

Conversation

thecodacus commented Nov 16, 2024 • edited Loading

Add Code Template System and Tool Calling Infrastructure

Overview

Key Changes

1. Template System Implementation

2. Tool Calling Infrastructure

3. Agent System

4. UI Enhancements

Technical Implementation

Agent Architecture

Template System

Tool Execution Pipeline

Testing

Migration Impact

Future Enhancements

Preview

thecodacus commented Nov 16, 2024

thecodacus commented Nov 16, 2024

wonderwhy-er commented Nov 16, 2024

thecodacus commented Nov 16, 2024

wonderwhy-er commented Nov 16, 2024

champ2050 commented Nov 17, 2024

thecodacus commented Nov 18, 2024

Prathamesh-Shanbhag commented Nov 22, 2024

wonderwhy-er commented Nov 23, 2024

thecodacus commented Nov 23, 2024 • edited Loading

adriano2023666 commented Nov 23, 2024

mrsimpson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Dec 6, 2024

thecodacus commented Nov 16, 2024 •

edited

Loading

thecodacus commented Nov 23, 2024 •

edited

Loading