Beyond Text Generation: Coding Ollama Function Calls and Tools

functions, tools, Ollama
Page content

Summary

Function calling allows Large Language Models (LLMs) to interact with APIs, databases, and other tools, making them more than just text generators.

Integrating LLMs with functions enables you to harness their powerful text processing capabilities, seamlessly enhancing the technological solutions you develop.

This post will explain how you can call local python functions and tools in Ollama.

Introduction to Ollama Function Calling

Ollama allows you to run state-of-the-art LLMs like Qwen, Llama, and others locally without relying on cloud APIs. Its function-calling feature enables models to execute external Python functions, making it ideal for applications like chatbots, automation tools, and data-driven systems.

Key benefits:

  • Local Execution: No API costs or internet dependency.
  • Function Integration: Easily integrate custom Python functions into your prompts.
  • Flexibility: Use any Python library or script as part of your model’s workflow.

Setting Up Ollama

Before diving into function calling, ensure you have Ollama installed and set up:

  1. Install Ollama: Follow the instructions on the Ollama website to install it on your machine.

  2. Download a Model:

    ollama pull llama3.2

  3. Test the Model:

    ollama run llama3.2 "Hello, how are you?"

I'm just a language model, so I don't have feelings or emotions like humans do. However, I'm functioning properly
and ready to assist you with any questions or tasks you may have! How can I help you today?

  1. Enable Function Calling: Ollama uses JSON-based communication for function calling. We will build an example application to send and receive these JSON messages.

  2. Why use function calls: There are lots of things LLM’s don’t do well or can’t do. We use the function calls to compliment tha ability of the LLM to enhance our complete solutions.

Basic Function Calling

Step 1: Define a Simple Function

Let’s start with a basic example where the LLM calls a Python function to calculate the square of a number.

Python Code:

import json
from flask import Flask, request, jsonify

app = Flask(__name__)

# Define a simple function
def square(x):
    return x * x

@app.route("/function", methods=["POST"])
def call_function():
    # Parse incoming JSON request
    data = request.json
    function_name = data.get("name")
    arguments = data.get("args", [])

    # Call the function dynamically
    if function_name == "square":
        result = square(*arguments)
        return jsonify({"result": result})
    else:
        return jsonify({"error": "Function not found"}), 404

if __name__ == "__main__":
    app.run(port=9999)

Explanation:

  • This Flask app defines a /function endpoint that accepts JSON requests.
  • The square function takes a single argument and returns its square.
  • The app dynamically calls the requested function based on the name field in the JSON payload.

Step 2: Configure Ollama Prompt

Use the following prompt format to call the square function:

{
  "functions": [
    {
      "name": "square",
      "description": "Calculates the square of a number",
      "parameters": {"type": "array", "items": {"type": "number"}}
    }
  ],
  "function_call": {"name": "square", "args": [5]}
}

Explanation:

  • The functions array defines available functions and their parameters.
  • The function_call object specifies which function to call and its arguments.

Step 3: Test the Function Call

Run the Flask app and test the function call using Ollama:

curl -X POST http://localhost:9999/function \
     -H "Content-Type: application/json" \
     -d '{"name": "square", "args": [5]}'

Output:

{"result": 25}

Advanced Function Calling

Example: Weather Forecasting

Suppose you want the LLM to fetch the current weather for a given location.

Python Code:

import requests
from flask import Flask, request, jsonify

app = Flask(__name__)

# Define a function to fetch weather data
def get_weather(location):
    api_key = "your_api_key_here"
    url = f"http://api.weatherapi.com/v1/current.json?key={api_key}&q={location}"
    response = requests.get(url)
    if response.status_code == 200:
        data = response.json()
        return data["current"]["temp_c"]
    else:
        return "Error fetching weather data"

@app.route("/function", methods=["POST"])
def call_function():
    data = request.json
    function_name = data.get("name")
    arguments = data.get("args", [])

    if function_name == "get_weather":
        location = arguments[0]
        temperature = get_weather(location)
        return jsonify({"temperature": temperature})
    else:
        return jsonify({"error": "Function not found"}), 404

if __name__ == "__main__":
    app.run(port=9999)

Explanation:

  • The get_weather function uses the WeatherAPI to fetch the current temperature for a given location.
  • The Flask app exposes this function via the /function endpoint.

Ollama Prompt:

{
  "functions": [
    {
      "name": "get_weather",
      "description": "Fetches the current weather for a location",
      "parameters": {"type": "array", "items": {"type": "string"}}
    }
  ],
  "function_call": {"name": "get_weather", "args": ["New York"]}
}

Test the Function Call:

curl -X POST http://localhost:9999/function \
     -H "Content-Type: application/json" \
     -d '{"name": "get_weather", "args": ["New York"]}'

Output:

{"temperature": 15}

Building a Complete Python App

Now, let’s build a complete Python app that integrates Ollama with function calling. This app will:

  1. Allow users to ask questions.
  2. Use Ollama to interpret the question.
  3. Call external Python functions when necessary.
  4. Return the result to the user.

Step 1: Install Required Libraries

pip install ollama requests flask

Step 2: Create the Flask Server

Save the following code as server.py:

import os
import requests
import logging
import logging.config
import json
from dotenv import load_dotenv
from flask import Flask, request, jsonify

# -------------------------------
# Load Environment Variables from .env File
# -------------------------------
load_dotenv()

# -------------------------------
# Configuration Class
# -------------------------------
class Config:
    API_KEY = os.getenv("WEATHER_API_KEY", "your_default_api_key")
    WEATHER_API_URL = os.getenv("WEATHER_API_URL", "http://api.weatherapi.com/v1/current.json")
    LOG_FILE = os.getenv("LOG_FILE", "server.log")
    LOG_LEVEL = logging.INFO
    PORT = int(os.getenv("PORT", 9999))

# -------------------------------
# Logging Configuration
# -------------------------------
LOGGING_CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "detailed": {
            "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
        }
    },
    "handlers": {
        "file": {
            "class": "logging.FileHandler",
            "filename": Config.LOG_FILE,
            "mode": "w",
            "formatter": "detailed"
        },
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "detailed"
        }
    },
    "root": {
        "handlers": ["file", "console"],
        "level": Config.LOG_LEVEL,
    }
}

logging.config.dictConfig(LOGGING_CONFIG)
logger = logging.getLogger(__name__)

# -------------------------------
# Flask App
# -------------------------------
app = Flask(__name__)

# -------------------------------
# Utility Functions
# -------------------------------
def square(x: int) -> int:
    """Returns the square of a number."""
    return x * x

def get_weather(location: str) -> dict:
    """Fetches the current temperature from WeatherAPI."""
    params = {"key": Config.API_KEY, "q": location}
    
    try:
        response = requests.get(Config.WEATHER_API_URL, params=params)
        response.raise_for_status()
        data = response.json()
        return {"temperature": data["current"]["temp_c"]}
    
    except requests.exceptions.RequestException as e:
        logger.error(f"Error fetching weather data: {e}")
        return {"error": "Failed to fetch weather data"}

# -------------------------------
# Flask Routes
# -------------------------------
@app.route("/function", methods=["POST"])
def call_function():
    """Handles function calls via POST requests."""
    try:
        data = request.get_json()
        logger.info(f"Received request: {json.dumps(data)}")

        if not data or "name" not in data or "args" not in data:
            logger.warning("Invalid request format")
            return jsonify({"error": "Invalid request format"}), 400

        function_name = data["name"]
        arguments = data["args"]

        if function_name == "square":
            if not arguments or not isinstance(arguments[0], (int, float)):
                return jsonify({"error": "Invalid argument for square"}), 400
            result = square(arguments[0])
            return jsonify({"result": result})

        elif function_name == "get_weather":
            if not arguments or not isinstance(arguments[0], str):
                return jsonify({"error": "Invalid argument for get_weather"}), 400
            temperature = get_weather(arguments[0])
            return jsonify(temperature)

        else:
            logger.warning(f"Function '{function_name}' not found")
            return jsonify({"error": "Function not found"}), 404

    except Exception as e:
        logger.error(f"Unexpected error: {e}")
        return jsonify({"error": "Internal server error"}), 500

# -------------------------------
# Run the Application
# -------------------------------
if __name__ == "__main__":
    app.run(port=Config.PORT)

Step 3: Create the Ollama Client

Save the following code as client.py:

import json
import requests

# Define the Ollama server URL
OLLAMA_URL = "http://localhost:11434/api/generate"

# Define the Flask server URL
FUNCTION_URL = "http://localhost:9999/function"

# Function to call Ollama
def call_ollama(prompt):
    headers = {"Content-Type": "application/json"}
    payload = {
        "model": "llama2",
        "prompt": prompt,
        "context": {
            "functions": [
                {"name": "square", "description": "Calculates the square of a number"},
                {"name": "get_weather", "description": "Fetches the current weather for a location"}
            ]
        }
    }
    response = requests.post(OLLAMA_URL, headers=headers, json=payload)
    return response.json()

# Function to call external functions
def call_external_function(function_name, args):
    headers = {"Content-Type": "application/json"}
    payload = {"name": function_name, "args": args}
    response = requests.post(FUNCTION_URL, headers=headers, json=payload)
    return response.json()

# Main interaction loop
if __name__ == "__main__":
    while True:
        user_input = input("Ask me anything: ")
        if user_input.lower() in ["exit", "quit"]:
            break

        # Check if the user wants to call a function
        if "call square" in user_input:
            try:
                number = int(user_input.split()[-1])
                result = call_external_function("square", [number])
                print(f"Result: {result['result']}")
            except Exception as e:
                print(f"Error: {e}")
        elif "call weather" in user_input:
            location = user_input.split("for")[-1].strip()
            result = call_external_function("get_weather", [location])
            print(f"Temperature in {location}: {result['temperature']}°C")
        else:
            # Pass the prompt to Ollama
            response = call_ollama(user_input)
            print("Ollama Response:", response["response"])

Step 4: Run the App

  1. Start the Flask server:

    python server.py

  2. Start the Ollama client:

    python client.py

  3. Interact with the app:

    Ask me anything: What is the square of 7?
Result: 49

Ask me anything: What is the weather for London?
Temperature in London: 12°C

Ask me anything: Tell me about AI.
Ollama Response: Artificial intelligence (AI) refers to the simulation of human intelligence in machines...

Step-by-Step Breakdown

Step 1: Define Functions

In server.py, define the functions you want the LLM to call. For example:

  • square: Calculates the square of a number.
  • get_weather: Fetches the current weather for a location.

Step 2: Expose Functions via Flask

Use Flask to expose these functions as endpoints. The /function route handles incoming requests and executes the appropriate function.

Step 3: Configure Ollama Prompts

When interacting with Ollama, include a context section in your prompts to specify available functions. For example:

"context": {
    "functions": [
        {"name": "square", "description": "Calculates the square of a number"},
        {"name": "get_weather", "description": "Fetches the current weather for a location"}
    ]
}

Step 4: Detect Function Calls

In client.py, detect keywords in the user input (e.g., “call square” or “call weather”) and invoke the corresponding external function.

Step 5: Handle Responses

Pass the results of external function calls back to the user or incorporate them into the LLM’s response.

Tools

Since version 0.4 the Ollama python library has added the ability to call tools. Essentially we pass the function reference to our chat call. The Ollama Python library uses Pydantic and docstring parsing to generate the JSON schema.


from ollama import ChatResponse, chat


def add_two_numbers(a: int, b: int) -> int:
  """
  Add two numbers

  Args:
    a (int): The first number
    b (int): The second number

  Returns:
    int: The sum of the two numbers
  """

  # The cast is necessary as returned tool call arguments don't always conform exactly to schema
  # E.g. this would prevent "what is 30 + 12" to produce '3012' instead of 42
  return int(a) + int(b)


def subtract_two_numbers(a: int, b: int) -> int:
  """
  Subtract two numbers
  """

  # The cast is necessary as returned tool call arguments don't always conform exactly to schema
  return int(a) - int(b)


# Tools can still be manually defined and passed into chat
subtract_two_numbers_tool = {
  'type': 'function',
  'function': {
    'name': 'subtract_two_numbers',
    'description': 'Subtract two numbers',
    'parameters': {
      'type': 'object',
      'required': ['a', 'b'],
      'properties': {
        'a': {'type': 'integer', 'description': 'The first number'},
        'b': {'type': 'integer', 'description': 'The second number'},
      },
    },
  },
}

messages = [{'role': 'user', 'content': 'What is three plus one?'}]
print('Prompt:', messages[0]['content'])

available_functions = {
  'add_two_numbers': add_two_numbers,
  'subtract_two_numbers': subtract_two_numbers,
}

response: ChatResponse = chat(
  'llama3.2',
  messages=messages,
  tools=[add_two_numbers, subtract_two_numbers_tool],
)

if response.message.tool_calls:
  # There may be multiple tool calls in the response
  for tool in response.message.tool_calls:
    # Ensure the function is available, and then call it
    if function_to_call := available_functions.get(tool.function.name):
      print('Calling function:', tool.function.name)
      print('Arguments:', tool.function.arguments)
      output = function_to_call(**tool.function.arguments)
      print('Function output:', output)
    else:
      print('Function', tool.function.name, 'not found')

# Only needed to chat with the model using the tool call results
if response.message.tool_calls:
  # Add the function response to messages for the model to use
  messages.append(response.message)
  messages.append({'role': 'tool', 'content': str(output), 'name': tool.function.name})

  # Get final response from model with function outputs
  final_response = chat('llama3.2', messages=messages)
  print('Final response:', final_response.message.content)

else:
  print('No tool calls returned from model')

Prompt: What is three plus one?
Calling function: add_two_numbers
Arguments: {'a': 3, 'b': 1}
Function output: 4
Final response: The answer to the equation 3 + 1 is 4.

Calling an external library

Here we are using requests to get the html content of a web page.

import ollama
import requests

available_functions = {
  'request': requests.request,
}

response = ollama.chat(
  'llama3.2',
  messages=[{
    'role': 'user',
    'content': 'get the programmer.ie webpage?',
  }],
  tools=[requests.request], 
)

for tool in response.message.tool_calls or []:
  function_to_call = available_functions.get(tool.function.name)
  if function_to_call == requests.request:
    # Make an HTTP request to the URL specified in the tool call
    resp = function_to_call(
      method=tool.function.arguments.get('method'),
      url=tool.function.arguments.get('url'),
    )
    print(resp.text)
  else:
    print('Function not found:', tool.function.name)

prints the html for the website.

Problems with this approach

I think it is important that we look objectively at our solutions.

  • As you may have noticed we need to intercept the chat to insert the functions.
  • This leads to a text processing solution like if the query starts with get me the weather then call the weather function.
  • Handling function names, arguments, and responses requires additional parsing, adding complexity to the implementation.
  • Implementing a solution that allows for dynamic function calls and parameter matching is challenging.
  • It’s a brittle solution. If we add 100 new functions, how will we manage which function was called?.
  • There is no qualitative assessment of whether executing a function is necessary or appropriate based on the context provided by the user’s query and other relevant data.

Challenges with Scalability

As the number of functions grows, managing them becomes increasingly complex. Hardcoding function names and arguments in the client can lead to brittle solutions. To address this:

  • Dynamic Function Discovery: Use a registry or configuration file to dynamically load available functions.

# -------------------------------
# Function Registry
# -------------------------------
FUNCTIONS = {}

def register_function(func):
    """Decorator to register a function in the function dispatcher."""
    FUNCTIONS[func.__name__] = func
    return func

# -------------------------------
# Registered Functions
# -------------------------------
@register_function
def square(x: int) -> int:
    """Returns the square of a number."""
    return x * x

@register_function
def get_weather(location: str) -> dict:
    """Fetches the current temperature from WeatherAPI."""
    params = {"key": Config.API_KEY, "q": location}

    try:
        response = requests.get(Config.WEATHER_API_URL, params=params)
        response.raise_for_status()
        data = response.json()
        return {"temperature": data["current"]["temp_c"]}

    except requests.exceptions.RequestException as e:
        logger.error(f"Error fetching weather data: {e}")
        return {"error": "Failed to fetch weather data"}

# -------------------------------
# Function Dispatcher
# -------------------------------
def call_registered_function(function_name, arguments):
    """Dynamically calls a registered function."""
    if function_name not in FUNCTIONS:
        return {"error": f"Function '{function_name}' not found"}, 404

    func = FUNCTIONS[function_name]
    sig = inspect.signature(func)

    try:
        # Convert arguments to match function signature
        bound_args = sig.bind(*arguments)
        bound_args.apply_defaults()
        
        result = func(*bound_args.args)
        return {"result": result}, 200

    except TypeError as e:
        logger.error(f"Error calling function '{function_name}': {e}")
        return {"error": f"Invalid arguments for function '{function_name}'"}, 400

# -------------------------------
# Flask Route for Function Calls
# -------------------------------
@app.route("/function", methods=["POST"])
def call_function():
    """Handles function calls dynamically via POST requests."""
    try:
        data = request.get_json()
        logger.info(f"Received request: {json.dumps(data)}")

        if not data or "name" not in data or "args" not in data:
            return jsonify({"error": "Invalid request format"}), 400

        function_name = data["name"]
        arguments = data["args"]

        response, status_code = call_registered_function(function_name, arguments)
        return jsonify(response), status_code

    except Exception as e:
        logger.error(f"Unexpected error: {e}")
        return jsonify({"error": "Internal server error"}), 500
  • Intent Detection: Leverage embeddings or NLP models to identify the most relevant function based on the user’s query.
  • Advanced Orchestration: Tools like LangChain or custom orchestrators can streamline function management and improve scalability.

Security Considerations

When exposing functions to LLMs, security must be a top priority:

  • Input Validation: Always validate user inputs to prevent injection attacks or unintended behavior.
  • Access Control: Restrict access to sensitive functions by requiring authentication or authorization.
  • Logging and Monitoring: Implement logging to track function calls and monitor for suspicious activity.

Conclusion

Ollama’s function-calling capability bridges the gap between natural language processing and real-world applications. By integrating external Python functions, you can empower LLMs to perform tasks like mathematical calculations, weather forecasting, and more. Its important to recognize the challenges and limitations of the presented approach. While this implementation works, it has scalability challenges. Future approaches might include dynamic function mapping, using embeddings for better intent detection, or integrating Ollama with more advanced orchestration tools.

Code examples

You can find the code for this post here

ollama-functions

References