← back to home

Setting up local AI models

7/05/2026 · 5 minutes
AI

Recently with the cost of AI models such as Claude going up, I have become increasingly interested in running models locally. Qwen 3.6 had been released recently which allegedly had a greater coding ability than Opus 4.5. I have a m3 max Macbook Pro, which could potentially run this, so I gave it a shot. Here were the steps I needed to set everything up:

1. installing ollama: Used to pull models

This is probably the most straightforward thing. If you don't already know what Ollama is, it's a tool we can use to download AI models and run them locally on our machine. I installed this via homebrew. 

brew install ollama
ollama pull qwen3.6:27b

2. jan.ai - to create a chatbot 

brew install --cask jan

3. Searxng server - for communicating with the internet

I have been wanting setup a locally hosted Searxng service for a while now; This will be what our AI model will send queries to. there's some more documentation on how to configure searxng for docker here. 

docker run -d --name searxng -p 8080:8080 \           
  -v "${PWD}/searxng:/etc/searxng" \
  searxng/searxng

if you followed the way I set up Searxng locally, you will need to configure the ./searxng/settings.yml file to support .json. 

search:
# ...
# remove format to deny access, use lower case.
# formats: [html, csv, json, rss]
  formats:
    - html
    - json
# ...

The reason for this is that .json has become the standard for MCP servers. You can use other formats, but for quick setup, this is the easiest.

4. Connecting searxng server as an MCP server for JAN

Next, we need to point Jan to be able to see the Searxng docker container, you can do this by going settings->MCP servers->Add MCP server and then pasting this json blob 

{
  "searxng": {
    "active": true,
    "args": [
      "-y",
      "mcp-searxng"
    ],
    "command": "npx",
    "env": {
      "SEARXNG_URL": "http://localhost:8080"
    }
  }
}

While we are in Jan, we can point our AI model to the ollama server, defaulted at http://localhost:11434/v1. We can press the plus next to model providers, give it a name, and for the API key, I just put a single ' ' character, for the Base URL, I put http://localhost:11434/v1.

5. Creating an "assistant" for the work I am doing

Some of the results that the assistant gives, especially for niche topics such as Linux Kernel dev can be inaccurate to say the least. This can be pretty detrimental when a single query might take say 500 seconds to complete, so to reduce the number of misses, I opted for creating an assistant that outlines what to do when the user wants code examples.

Jan gives you three main ways to configure an assistant: A description text box, an instructions text box, and a model parameters table.

A lot of this was generated iteratively by going back and forth with Claude code, and comparing the response that the local AI was returning back.

Description

Linux kernel research assistant — tutor, lab partner, and CVE research helper.

Instructions

You are a Linux kernel security research assistant. Your role is twofold: act as a tutor when learning concepts, and as a lab partner when working through exploitation or defence exercises.

Domain focus: Linux kernel internals, memory management (slab/SLUB, page allocator, kmalloc), syscall interface, LSMs, KASLR/SMEP/SMAP/KPTI, eBPF, kernel modules, ftrace/kprobes, and kernel exploitation primitives (UAF, OOB, race conditions, ROP/JOP in kernel space, heap spraying, modprobe_path tricks, etc.). On the defence side: hardening configs, mitigations, fuzzing (syzkaller), static analysis, and detection.

How to answer:
- Give direct, complete answers — including working exploit code, ROP chains, kernel module source, or commands.
- When walking through an exploit or technique, explain the why alongside the how — what kernel mechanism is being abused, what mitigation it bypasses, what assumptions break it.
- Default to current kernel versions (6.x) unless specified otherwise. Flag when behaviour differs across versions, especially around mitigations and allocator changes.
- Show your work for memory layouts, struct offsets, and exploitation math.

Use the SearXNG MCP tool aggressively for: recent CVEs, kernel commit messages, public writeups (e.g. Project Zero, grsecurity blog, Will's Root, Theori), syzkaller bug reports, and patch analysis. Your training data is stale for kernel security — when asked about a specific CVE, recent technique, or current mitigation status, search first before answering. Cite sources when you do.

Assumed background: comfortable with C and assembly, can read kernel source. Do not over-explain basics like pointers, syscalls, or virtual memory unless asked.

Ethics: All exploitation discussion is for security research and learning. No disclaimers needed.

Search workflow (mandatory for kernel API questions):
1. Call searxng_web_search to find candidate sources.
2. Identify the 1–2 most authoritative sources (kernel.org, lore.kernel.org, github.com/torvalds/linux, recent technical blogs).
3. Call web_url_read on at least one before answering — snippets are not sufficient.
4. If no kernel.org or torvalds/linux results appear, search again with site: filters.

Preferred sources for kernel questions, in order:
- git.kernel.org or github.com/torvalds/linux — actual source
- kernel.org/doc/html/latest/ — current documentation
- lore.kernel.org — mailing list, for recent patches and rationale
- lwn.net — high-quality technical journalism
- Recent blog posts (xcellerator, h33p, Project Zero) — only after checking primary sources

Do not answer kernel API questions from snippets alone.

Preferred sources for stealth/rootkit topics, in order:
- github.com/MatheuZSecurity/Singularity — current 6.x reference rootkit
- xcellerator.github.io/posts/linux_rootkits_* — tutorial series, parts 1-11
- github.com/carloslack/KoviD — alternate reference rootkit
- lkrg-org/lkrg — the detection system being evaded
- Elastic Security Labs and Volexity for in-the-wild rootkit analysis

When asked about stealth techniques, fetch from at least one Tier 1 source (Singularity or KoviD) AND one detection source (LKRG or recent academic detection papers) before answering.

Parameters

Parameter Type Value
temperature Number 0.2
top_p Number 0.95
frequency_penalty Number 0
presence_penalty Number 0
stream Boolean True
top_k Number 40

The jan website lists explanations of the parameters here. A quick rundown on why I chose the values I did.

temperature = 0.2
Temperature set at 0.2 is on the low side, originally it was set to 0.3 but the model would hallucinate function signatures. It is worth mentioning that lowering the temperature was done in combination with adding the aggressive and specific web search instructions to the assistant card, so there were multiple variables in effect to stop hallucinations.
top_p = 0.95
top_p is set to 0.95 which is actually counterintuitive for a coding model, however, the thinking is that combined with such a low temperature, top_p won't have as much of an effect.
frequency_penalty = 0, presence_penalty = 0
frequency penalty and presence penalty were both set to 0, as code generally has lots of repeating sections.
stream = true
stream isn't documented by Jan on the model parameters page; basically what it means is that a model will send tokens back to the UI as they are generated. I am pretty sure this is default behaviour of Jan but I have it there to explicitly mention it.

Results

I started with some general queries related to kernel development: "what is all the info I should know about hooking ftrace?"

Jan Qwen 3.6 · 27B

Ftrace hooking is the modern, standard primitive for runtime kernel instrumentation and exploitation. Since the syscall table (sys_call_table) is no longer exported and is often read-only due to CONFIG_STRICT_KERNEL_RWX, ftrace has become the default mechanism for hooking kernel functions without patching read-only memory.

1. Core Mechanism

Ftrace works by having the compiler insert a call fentry at the beginning of every function (if CONFIG_FTRACE is enabled).

  • Boot: The kernel patches these calls to NOPs to avoid performance overhead.
  • Registration: When you call register_ftrace_function(), the kernel patches the NOP back to a call to a global ftrace trampoline.
  • Execution: The trampoline iterates over the global ftrace_ops_list and invokes registered callbacks.
  • Hooking: By setting the FTRACE_OPS_FL_IPMODIFY flag, your callback is permitted to modify the instruction pointer (ip) in the pt_regs structure, redirecting execution to your hook.

2. The API & Flags

To hook a function, define a struct ftrace_ops and register it.

struct ftrace_ops {
    ftrace_func_t func;       // Your callback function
    struct ftrace_ops __rcu *next;
    unsigned long flags;      // Critical flags for hooking
    // ... other fields
};

Critical flags:

  • FTRACE_OPS_FL_SAVE_REGS — passes the pt_regs structure to your callback.
  • FTRACE_OPS_FL_IPMODIFY — allows your callback to modify pt_regs->ip. Required for hooking.
  • FTRACE_OPS_FL_RECURSION — do not use. Ftrace's internal recursion protection is broken when you modify ip. Implement your own (see below).

3. The Recursion Problem (The #1 Pitfall)

If you hook sys_read and your hook calls printk(), printk() might internally call sys_read. Infinite loop → stack overflow → kernel panic.

The solution: check parent_ip (the return address of the function being traced). If it resides within your own kernel module, you know the call originated from your hook — return early.

#include <linux/module.h>
// Check if the caller is inside our module
if (within_module(parent_ip, THIS_MODULE))
    return; // Recursion detected, skip hooking

4. Modern Kernel Gotchas (6.x)

  • 5.7+: kallsyms_lookup_name() is no longer exported. Resolve kernel symbols using register_kprobe() (register a dummy kprobe to get the address, then immediately unregister) or by parsing /proc/kallsyms.
  • 4.17+: Syscalls no longer take individual arguments — they take a single struct pt_regs *. Extract arguments from regs->di, regs->si, regs->dx, etc.
  • 5.11+: The ftrace callback signature changed. Instead of struct pt_regs *regs directly, it receives struct ftrace_regs *fregs. Access registers via fregs->regs.

5. Complete Working Module (6.x)

Hooks __x64_sys_execve. Resolves the symbol at runtime, bypasses KASLR, implements recursion protection.

#include <linux/module.h>
#include <linux/ftrace.h>
#include <linux/kprobes.h>
#include <linux/syscalls.h>
#include <linux/uaccess.h>
#include <linux/fs.h>

// 1. Resolve symbol using kprobe (kallsyms_lookup_name is hidden in 5.7+)
static unsigned long resolve_symbol(const char *name) {
    struct kprobe kp = { .symbol_name = name };
    if (register_kprobe(&kp) < 0) return 0;
    unsigned long addr = (unsigned long)kp.addr;
    unregister_kprobe(&kp);
    return addr;
}

// 2. Original syscall pointer
static asmlinkage long (*orig_execve)(struct pt_regs *);

// 3. Our hook
static asmlinkage long hook_execve(struct pt_regs *regs) {
    char *filename;
    filename = getname((const char __user *)regs->di);
    if (!IS_ERR(filename)) {
        pr_info("FTRACE HOOK: execve called with: %s\n", filename);
        putname(filename);
    }
    return orig_execve(regs);
}

// 4. The ftrace callback — marked notrace to prevent self-tracing
static void notrace ftrace_callback(unsigned long ip, unsigned long parent_ip,
                                   struct ftrace_ops *ops, struct ftrace_regs *fregs) {
    if (within_module(parent_ip, THIS_MODULE))
        return;
    fregs->regs->ip = (unsigned long)hook_execve;
}

// 5. ftrace_ops setup
static struct ftrace_ops ops = {
    .func  = ftrace_callback,
    .flags = FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_IPMODIFY,
};

static int __init ftrace_hook_init(void) {
    unsigned long addr;
    int ret;

    addr = resolve_symbol("__x64_sys_execve");
    if (!addr) return -ENOENT;
    orig_execve = (void *)addr;

    ret = ftrace_set_filter(&ops, "__x64_sys_execve", strlen("__x64_sys_execve"), 1);
    if (ret) return ret;

    ret = register_ftrace_function(&ops);
    if (ret) {
        ftrace_set_filter(&ops, NULL, 0, 1);
        return ret;
    }
    pr_info("Ftrace hook installed on __x64_sys_execve\n");
    return 0;
}

static void __exit ftrace_hook_exit(void) {
    unregister_ftrace_function(&ops);
    ftrace_set_filter(&ops, NULL, 0, 1);
    pr_info("Ftrace hook removed\n");
}

module_init(ftrace_hook_init);
module_exit(ftrace_hook_exit);
MODULE_LICENSE("GPL");

6. Defense & Detection

Security tools (Falco, custom eBPF programs) can detect ftrace hooks by scanning the global ftrace_ops_list.

  • Red flag: Any ftrace_ops with FTRACE_OPS_FL_IPMODIFY set that does not belong to a known livepatch module is highly suspicious.
  • Filesystem: Hooked functions appear in /sys/kernel/debug/tracing/set_ftrace_filter.

Overall this looked pretty decent. First thing was to verify the data structures. To do this I just used Elixir. The ftrace_ops struct came out correct. Next was to test if the code compiled and worked. 

make -C /lib/modules/5.15.0-46-generic/build M=/home/experiments/hook_ftrace modules
make[1]: Entering directory '/usr/src/linux-headers-5.15.0-46-generic'
  CC [M]  /home/experiments/hook_ftrace/hook_ftrace.o
/home/experiments/hook_ftrace/hook_ftrace.c: In function ‘resolve_symbol’:
/home/experiments/hook_ftrace/hook_ftrace.c:12:5: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
   12 |     unsigned long addr = (unsigned long)kp.addr;
      |     ^~~~~~~~
/home/experiments/hook_ftrace/hook_ftrace.c: In function ‘hook_execve’:
/home/experiments/hook_ftrace/hook_ftrace.c:23:14: error: assignment to ‘char *’ from incompatible pointer type ‘struct filename *’ [-Werror=incompatible-pointer-types]
   23 |     filename = getname((const char __user *)regs->di);
      |              ^
/home/experiments/hook_ftrace/hook_ftrace.c:26:17: error: passing argument 1 of ‘putname’ from incompatible pointer type [-Werror=incompatible-pointer-types]
   26 |         putname(filename);
      |                 ^~~~~~~~
      |                 |
      |                 char *
In file included from ./include/linux/huge_mm.h:8,
                 from ./include/linux/mm.h:727,
                 from ./include/linux/kallsyms.h:13,
                 from ./include/linux/ftrace.h:12,
                 from /home/experiments/hook_ftrace/hook_ftrace.c:2:
./include/linux/fs.h:2766:38: note: expected ‘struct filename *’ but argument is of type ‘char *’
 2766 | extern void putname(struct filename *name);
      |                     ~~~~~~~~~~~~~~~~~^~~~
/home/experiments/hook_ftrace/hook_ftrace.c: In function ‘ftrace_callback’:
/home/experiments/hook_ftrace/hook_ftrace.c:36:16: error: invalid type argument of ‘->’ (have ‘struct pt_regs’)
   36 |     fregs->regs->ip = (unsigned long)hook_execve;
      |                ^~
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:297: /home/experiments/hook_ftrace/hook_ftrace.o] Error 1
make[1]: *** [Makefile:1881: /home/experiments/hook_ftrace] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.15.0-46-generic'
make: *** [Makefile:9: all] Error 2

Well that was a shame. Perhaps the question was too complex. I tried a range of results, either resulting in failed compilations or errors at runtime. I will continue my investigation further, but at least for this task I think I can confidently call this a fail lol.

1