AWS Bedrock Agent

In this post we show creation of Agent in AWS bedrock and a simple text chat application using python boto3 library.

Create Agent in AWS Console

First open the AWS console, navigate to the AWS bedrock service, and click on Agents.

Click on Create Agent, enter a name for it, and click Create.

For our demo, we switch to the cheapest available model:

Fill in instructions for the agent:

Now click on Save and Exit.

Click on Prepare to create a draft of the agent:

We can now test the agent is working as expected, or else update the agent instructions.

Now we create an alias for the agent, which is a published version of the agent.

Chat using python boto3

Use the following code to run a simple chat with the agent.

import uuid

import boto3

REGION = "us-east-1"
AGENT_ID = "AB5BQ7PVAL"
AGENT_ALIAS_ID = "9BII1XBJP9"

client = boto3.client('bedrock-agent-runtime', region_name=REGION)

session_id = str(uuid.uuid4())

print("Chat Starting")

while True:
  user_input = input("Enter prompt:")
  if user_input.lower() in ['exit', 'quit']:
    print("Bye")
    break

  response = client.invoke_agent(
    agentId=AGENT_ID,
    agentAliasId=AGENT_ALIAS_ID,
    sessionId=session_id,
    inputText=user_input,
  )

  response_body = response['completion']

  print("Answer:", end=" ", flush=True)
  for chunk in response_body:
    if 'chunk' in chunk:
      print(chunk['chunk']['bytes'].decode("utf-8"), end="", flush=True)
  print()  

An example of this chat is below.

Final Note

This is a very simple example of the AWS bedrock agent activation. Other agent properties include guardrails, agent memory across sessions, and multi-agent configuration.

Best practices for agents creation can be found here.

GO Embed

 

In this post we review the GO embed annotation and its implications.

Sample Code

package main

import (
  "embed"
  _ "embed"
  "fmt"
)

//go:embed hello.txt
var textFile string

//go:embed hello.txt
var binaryFile []byte

//go:embed data1 data2
var files embed.FS

func main() {
  fmt.Printf("%v bytes:\n%v\n", len(binaryFile), textFile)

  entries, err := files.ReadDir(".")
  if err != nil {
   panic(err)
  }
  for _, entry := range entries {
   fmt.Printf("%v dir: %v\n", entry.Name(), entry.IsDir())
  }
}

Implications

The GO embed is a simple way of adding files as part of the GO compiled output binary. It serves as an aletnative to making this files available to the application in other mannger, such as supplying the files as part of a docker image, or mounting the files using a kubernetes ConfigMap.

Notice the files are added as part of the binary, so embedding large files means a larger output binary.

Embed Methods

The are 3 methods to embed a file.

First we can add a file as a string. In such a case we should add the explicit embed import:

_ "embed"

Second we can add the file as bytes array, this is very similar to the first method.

Third we can include a set of folders as a virtual file system. The annotation includes the list of folders to be included. There are special handling for files starting with a dot, see more about this in here.

Final Note

While embed is a simple way to add files, it should be used only if we're sure we will not want to change the files in an active running deployment.

How To Improve LLM Inference Performance

 

In this post we will review possible changes to a LLM inference code to make it run faster and use less GPU memory.

LLM inference is the usage of a trained model for a new data and producing a classification or a prediction. This is usually the production time usage of the model we've selected and possibly fine-tuned for the actual data stream. 

The inference runs the following steps:

1. Load the model to the memory once in the process startup
2. Get an input of a single or preferably a batch of inputs 
3. Forward calculation on the neural network and produce a result

The LLM performance term is actually used for both different subjects:

1. The precision of the LLM such as false-positive, false-negative
2. The GPU memory usage and time for the inference process

We will see later that while these are two different goals, they are actually intertwined.

Below is a sample code of model inference where we can see the model loading and the model inference.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("ProtectAI/deberta-v3-base-prompt-injection-v2")
model = AutoModelForSequenceClassification.from_pretrained("ProtectAI/deberta-v3-base-prompt-injection-v2")

classifier = pipeline(
  "text-classification",
  model=model,
  tokenizer=tokenizer,
  truncation=True,
  max_length=512,
  device=torch.device(torch_device),
)

results = classifier(text)

# analyze the results
...

Let examine how can we improve the performance of the inference

Compile The Model

The first and simplest change is to compile the model:

model = torch.compile('cuda')

This simple instruction speeds up the inference up to 2 times faster!

The compile command coverts the mode python code to a compiled code and optimizes the model operations. Notice that the compile command should run only once, right after the model is loaded, and it has a small impact on the process startup time.

For the compile() instruction we will need to make sure we have both python-dev and g++ installed. An example of this in a Dockerfile is:

RUN apt-get update && \
    apt-get install -y software-properties-common curl wget build-essential g++ && \
    add-apt-repository ppa:deadsnakes/ppa && \
    apt-get update && \
    apt-get install -y python3.12 python3.12-dev && \
    /usr/bin/python3.12 --version

Change The Model Precision

By default the model is using a float32 precision, which means it has full accuracy in the neural network calculations. In most cases using float16 precision would do the work just as well and will consume half of the time and half of the memory. The change from float32 to float16 is called half().

model = model.to('cuda').half()

(Notice that model.half should be called BEFORE the model.compile)

Using half() might case a small increase in the false-positive and false-positive rate, but in most cases it is negligible. 

As a side note, we should mention that we can set the precision to int8 or int4, but this is a less common practice. For the record, here is an analyze of the alternatives from GPT:

Final Note

We have reviewed method of improving the memory footprint and runtime of an LLM. While there are some small implications on the accuracy, these methods should be used as a common practice for any LLM implementation.

NPX

 

In this post we will review NPX the Node Package Execute tool.

NPX is a command line util that is installed as part of the Node installation. Notice that this means that npx version is coupled with the node version.

The NPX temporary installs of packages that are used for a "script like"execution of a package. Instead of using npm to globally install a package and then run it, npx handles both.

The first step of NPX is to download the required package and its dependencies. The download target is:

~/.npm/_npx/<HASH>

The hash is based on the name and version of the package that NPX runs. Notice that the folder is never automatically removed, so once downloaded it will not be re-downloaded, that is unless we manually remove the cache folder or run a different version.

NPX however will not download to the cache folder if the package already exists in the current project node_modules folder, or if the package is globally installed.

To force NPX to download the latest version of a package and ignore any local or global installed version, we should specify the NPX flag --ignore-existing.

Common usages of NPX are:

npx create-react-app my-app
This will set the skeleton of a new react based application.

npx serve
This runs a file server on the current folder, enabling a quick review of the HTML files using a browser.

By default the NPX downloads the package, and the checks for "bin" entry in the package.json, which specifies the javascript file name to run. However we can manually determine the command to run using the syntax:

npx --package my-package my-command

In such case NPX would download the package and then look for the command under the bin element in package JSON and run it. Notice that we cannot run any javascript file using NPX, but only the predefined entries in the bin element. We can however, download the package using NPX, and the run any javascript file using node from the local NPX cache folder.

Create GO MCP Server and a Valid SSL Certificate

 

In this post we create GO bassed MCP server. 

The server includes both SSE and Streaming support. Gemini CLI supports only SSE, but the standard seems to be going to HTTM Streaming protocol.

We support both HTTP server on port 80, and HTTPS server on port 443. When using HTTPS server on port 443, we also start a certificate management listener on port 80 which will allocate a valid certificate as long as we have a valid DNS pointing to our server. Notice that for the SSL certification allocated we need to allow all source IPs to the port 80 listener, since the request will arrive from the Lets' encrypt servers.

package main

import (
  "context"
  "encoding/json"
  "fmt"
  "net/http"

  "github.com/modelcontextprotocol/go-sdk/jsonschema"
  "github.com/modelcontextprotocol/go-sdk/mcp"
  "golang.org/x/crypto/acme/autocert"
)

func main() {
  implementation := mcp.Implementation{
   Name: "Demo MCP Server",
  }
  mcpServer := mcp.NewServer(&implementation, nil)

  mcpServer.AddTool(toolSchema(), toolExecute)

  useSse := false
  var mcpHandler http.Handler
  if useSse {
   mcpHandler = mcp.NewSSEHandler(func(*http.Request) *mcp.Server {
    return mcpServer
   })
  } else {
   mcpHandler = mcp.NewStreamableHTTPHandler(func(request *http.Request) *mcp.Server {
    return mcpServer
   }, nil)
  }

  useTls := true
  if useTls {
   certManager := autocert.Manager{
    Prompt:     autocert.AcceptTOS,
    HostPolicy: autocert.HostWhitelist("my.domain.com"),
    Cache:      autocert.DirCache("certs"),
   }

   httpServer := &http.Server{
    Addr:      "0.0.0.0:443",
    Handler:   mcpHandler,
    TLSConfig: certManager.TLSConfig(),
   }

   go func() {
    err := http.ListenAndServe("0.0.0.0:80", certManager.HTTPHandler(nil))
    if err != nil {
     panic(err)
    }
   }()

   err := httpServer.ListenAndServeTLS("", "")
   if err != nil {
    panic(err)
   }
  } else {
   err := http.ListenAndServe("0.0.0.0:80", mcpHandler)
   if err != nil {
    panic(err)
   }
  }
}

Now we implement the tool that can both read input parameters and return a string result for the AI agent to send to the LLM.

type inputParameters struct {
  Name string `json:"name"`
}

func toolSchema() *mcp.Tool {
  return &mcp.Tool{
   Name:        "greet",
   Description: "Say hi from me",
   InputSchema: &jsonschema.Schema{
    Type:     "object",
    Required: []string{"name"},
    Properties: map[string]*jsonschema.Schema{
     "name": {
      Type: "string",
     },
    },
   },
  }
}

func toolExecute(
  _ context.Context,
  _ *mcp.ServerSession,
  params *mcp.CallToolParamsFor[map[string]any],
) (
  *mcp.CallToolResultFor[any],
  error,
) {
  bytes, err := json.Marshal(params.Arguments)
  if err != nil {
   panic(err)
  }

  var input inputParameters
  err = json.Unmarshal(bytes, &input)
  if err != nil {
   panic(err)
  }

  content := mcp.TextContent{
   Text: fmt.Sprintf("Hi %v, Demo MCP is at your service", input.Name),
  }

  result := mcp.CallToolResultFor[any]{
   Content: []mcp.Content{
    &content,
   },
  }

  return &result, nil
}

Create a Python MCP Server

 

MCP  is the standard protocol for exposing tools for AI agents usage. The MCP exposes APIs and the documentation for each API to the LLM. In this post we create a simple python based MCP server and use it in Gemini CLI.

Create The MCP Server

To prepare the MCP server project use:

# install UV in case it is not already installed
curl -LsSf https://astral.sh/uv/install.sh | sh

uv init magic_server
cd magic_server
uv venv
uv add "mcp[cli]" 

Next we add our main.py and expose our tool:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("Demo")

@mcp.tool(name="do_magic_trick")
def do_magic_trick(a: int, b: int) -> int:
 return a + b + 6

mcp.run(transport="stdio")
#mcp.run(transport="streamable-http")
#mcp.run(transport="sse")

We can run the MCP as a local tool using STDIO for the lower level communication, or using SSE and streamable-http to connect to the server over the network.

Run Gemini CLI and MCP Server STDIO

To use gemini CLI, make sure you have recent version of node, and run:

npx https://github.com/google-gemini/gemini-cli

Run the MCP as a local tool using the STDIO. To configure this, update the file ~/.gemini/settings.json

{
  "selectedAuthType": "oauth-personal",
  "mcpServers": {
    "pythonTools": {
      "command": "/home/my-user/.local/bin/uv",
      "args": [
        "run",
        "main.py"
      ],
      "cwd": "/home/my-user/magic_server",
      "env": {
      },
      "timeout": 15000
    }
  }
}

Once we restart the gemini CLI it will run the MCP server to get the metadata of the exposed tools, and we can use the tools simple by using the prompt: "do the magic_trick for 5 and 6".

Run Gemini CLI and MCP Server Service

To expose the MCP server as a public tool, we need to run the MCP server over the network. For this we can use SSE and http-streaming, so update the main.py to use the relevant method.

Notice:

1. At this time gemini CLI supports only SSE, but it seems that http-streaming is the winning standard.

2. In case using TLS (HTTPS access) the server should use a valid TLS certificate.

To configure the gemini to use the tool over the network, update the file ~/.gemini/settings.json:

{
  "selectedAuthType": "oauth-personal",
  "mcpServers": {
    "discoveredServer": {
      "url": "http://localhost:8000/sse"
    }
  }
}

Unlike running MCP using the STDIO, in this case we need to run the MCP ourselves:

uv run main.py

UV

 

In this post we will review the uv - a new python project manager.

I am not a heavy python user. I generally avoid using python for long-term existing projects as its maintainability is much complex due to its limited variable typing, and due to its single core usage. I usually use python for very short lived projects or for LLMs which get support almost only in python. Over the years I got used to all python pains: 

• Installation and usage the correct version of python and pip
• Creation of the python VENV
• Dependencies management using the requirement.txt which somehow never works

And then, about a year ago, a new tool emerged: uv.

The uv provides a complete solution for the entire python project management. It includes:

• Python version installation and management
• Dependencies add and locking
• New project creation
• VENV management
• Running helper tools

The funny thing about uv is that it is written in RUST, which is in my opinion a kind of an insult to python.

Anyways, listed below are some basic uv actions.

Create A New Project

To create a new project run the following commands.

mkdir demo
cd demo
uv init
uv python list
uv python pin 3.12
uv env

In case using PyCharm, configure it to use uv:

(make sure you have latest version of PyCharm)

PyCharm Settings --> Python --> Interpreter --> select the existing from the .venv

Dependencies

Adding dependecies is simple, for example, add flask.

uv add flask
uv lock  

Notice the uv.lock is automatically updated upon any additional add of dependency.

Unit Test and Converage

To run unit test and converage, add the dependencies as DEV dependencies, and then run the related tests.

uv add --dev pytest coverage
uv run -m coverage run -m pytest
uv run -m coverage report

Example of tests are below.

main.py

def add(a, b):
  return a + b

test_main.py

from main import add

def test_add():
  assert add(2, 3) == 5