Inside NESH

Overview

What NESH is from the inside, the principles that shaped it, and a map of this manual.

What this manual covers

The user manual explains what NESH does. This manual explains how. After reading it you should be able to find your way in every source file, build and test the shell, add commands and BASIC functions, and change the firmware-facing parts without breaking the others. It assumes you know C; UEFI concepts are explained where they are needed, and the glossary summarizes them.

Design principles

The project in numbers

The resulting nesh.efi is about 360 KB. The line counts here and in the source file map are not typed by hand: make docs counts the files and writes them into this page, and make test fails when they no longer match the tree — a new source file without a row in the map fails as well.

Source tree and tools

Repository layout

Tools needed

Make targets

Architecture

NESH is a stack of layers. Each layer only calls the layers below it, and everything that depends on the firmware is kept out of the portable core.

Layers

flowchart TB
  subgraph Portable["Portable (UEFI and Linux builds)"]
    CMD["Portable commands<br/>src/cmd"]
    BASIC["NESH BASIC<br/>src/basic"]
    CORE["Shell core<br/>src/core"]
    LIB["Runtime library<br/>src/lib"]
  end
  subgraph EFI["UEFI only"]
    PLAT["UEFI modules<br/>src/platform/efi"]
    PALE["pal_efi.c"]
  end
  subgraph HOST["Linux only"]
    PALH["pal_host.c + host.c"]
  end
  CMD --> CORE
  BASIC --> CORE
  PLAT --> CORE
  CORE --> PAL["pal.h interface"]
  PAL --> PALE
  PAL --> PALH
  CORE --> LIB
  PALE --> FW["UEFI firmware services"]

• Runtime library — the C functions everybody uses. In the UEFI build they are implemented by NESH (libc.c, fmt.c); in the Linux build rt.h includes the system headers instead.
• PAL (pal.h) — console, time, files, volumes, reset. The only way the portable code reaches the machine.
• Shell core — command registry and execution, output sinks, environment, aliases, line editor.
• BASIC — the language; it runs commands through the core.
• Commands — portable ones in src/cmd; UEFI ones in src/platform/efi, which also register BASIC functions and implement the platform hooks declared by the core (platform_run_image, platform_env_store, secure_boot_active…). In the Linux build src/platform/host.c implements the same hooks as stubs.

Start-up sequence

sequenceDiagram
  participant FW as UEFI firmware
  participant EM as efi_main (pal_efi.c)
  participant NM as nesh_main (shell.c)
  participant PC as platform_cmds_init
  FW->>EM: StartImage(nesh.efi)
  EM->>EM: globals gST/gBS/gRT, watchdog off,<br/>TSC calibration, volumes, arguments
  EM->>EM: allocate a 1 MiB stack and switch to it
  EM->>NM: nesh_main(argc, argv)
  NM->>NM: register BASIC functions and portable commands
  NM->>PC: platform_cmds_init()
  PC->>PC: install EFI_SHELL_PROTOCOL,<br/>register UEFI commands and functions
  NM->>NM: cd to the boot volume, env_init, alias_init
  NM->>NM: parse -n / -c / -k / SCRIPT
  NM->>NM: banner, startup.nsb, REPL
  NM-->>EM: exit code
  EM->>EM: efi_exit_hook: uninstall the shell protocol
  EM-->>FW: EFI status

Two details matter. First, UEFI only guarantees 128 KiB of stack and the interpreter is recursive, so efi_main allocates 1 MiB of pages and calls nesh_main on that stack (call_on_stack). Second, when NESH exits nothing may keep pointing into its image: efi_exit_hook uninstalls (or restores) the shell protocol.

EFI_STATUS EFIAPI efi_main(EFI_HANDLE image, EFI_SYSTEM_TABLE *st)
{
    gImage = image;
    gST = st;
    gBS = st->BootServices;
    gRT = st->RuntimeServices;
    gBS->SetWatchdogTimer(0, 0, 0, NULL);
    ...
    /* The interpreter is recursive: run on a private 1 MiB stack instead of
     * the firmware stack (UEFI only guarantees 128 KiB). */
    if (gBS->AllocatePages(AllocateAnyPages, EfiLoaderData, STACK_PAGES, &stack) == EFI_SUCCESS) {
        call_on_stack(main_on_stack, (void *)(uintptr_t)(stack + STACK_PAGES * 4096));
        ...
    if (efi_exit_hook)
        efi_exit_hook(); /* nothing may point into this image after it exits */
    return rc ? EFIERR(rc & 0xFF) : EFI_SUCCESS;
}

Life of a command line

flowchart TD
  L["line typed at the prompt"] --> B{"interp_is_basic_line?"}
  B -- yes --> I["interp_exec_interactive<br/>(may ask for more lines)"]
  B -- no --> S["shell_exec_line<br/>split words, extract > / >>"]
  S --> R{"redirection?"}
  R -- yes --> P["out_push_file"]
  R -- no --> X
  P --> X["shell_exec_argv"]
  X --> A{"alias?"}
  A -- yes --> X
  A -- no --> V{"fsN: alone?"}
  V -- yes --> CD["shell_chdir"]
  V -- no --> C{"built-in command?"}
  C -- yes --> D["strip -data, set data mode,<br/>call Cmd.fn(argc, argv)"]
  C -- no --> F["shell_find_executable<br/>(path, .nsb, .efi)"]
  F --> K{".nsb?"}
  K -- yes --> BR["basic_run_file"]
  K -- no --> EI["platform_run_image<br/>(efi_start_image)"]

Scripts take the same route: RUN "cmd" calls shell_exec_line, RUN "cmd", a$ calls shell_exec_argv directly (no word splitting), and RUN$ does the same inside a capture sink.

The build

NESH is built with the ordinary Linux GCC, linked as a position-independent ELF shared object, and converted to a PE32+ EFI application by a small Python tool.

Setting up a machine

The repository is self-contained: the sources, the tests with their binary fixtures, the documentation and the tools that generate it are all committed, and nothing is fetched at build time. A clone plus the packages below is the whole development environment; build/ holds only products and is not committed.

$ git clone https://github.com/nic-fio/NG-EFI_SHELL.git
$ cd NG-EFI_SHELL
$ tools/setup-dev.sh --install

How the project is worked on — the agreements with its author, the decisions that are settled, what to check before a commit and where the project stands — is in CLAUDE.md at the root of the repository. tools/backup.sh writes the whole repository, history and tags included, into a single bundle file that clones back without a network.

Pipeline

flowchart LR
  C["*.c"] -- "gcc -ffreestanding -fPIC<br/>-mno-red-zone -fshort-wchar" --> O["*.o"]
  O -- "ld -shared -Bsymbolic<br/>-T tools/efi.lds" --> SO["nesh.so<br/>(ELF, base 0)"]
  SO -- "tools/elf2efi.py" --> EFI["nesh.efi<br/>(PE32+, subsystem 10)"]
  C -- "gcc -DNESH_HOST<br/>-fsanitize=address,undefined" --> H["nesh-host<br/>(Linux)"]

Compiler flags and why

Warnings: -Wall -Wextra without -Wunused-parameter and -Wmissing-field-initializers. A clean build has no warnings; keep it that way.

The linker script

tools/efi.lds links at base address 0 with page-aligned .text, .rodata, .data (with the GOT) and .bss, and discards notes, comments and unwind tables. Base 0 makes ELF virtual addresses equal to PE relative virtual addresses, which keeps the conversion trivial. Page alignment gives each PE section its own memory protection (code read+execute, data read+write, never both).

elf2efi.py

The converter reads the ELF sections and writes a PE32+ image:

1. keeps .text, .rodata (as .rdata), .data and .bss, checking page alignment;
2. for every R_X86_64_RELATIVE relocation writes the addend into the section data and records a DIR64 fixup; any other relocation type is a build error (it means a symbol escaped the hidden visibility);
3. builds a .reloc section with one block per 4 KiB page;
4. writes the DOS stub, COFF header (machine 0x8664), optional header (subsystem 10 = EFI application, DllCharacteristics DYNAMIC_BASE | NX_COMPAT) and the section table.

Why not objcopy --target=efi-app-x86_64? It was tried first: it dropped sections and produced corrupted relocations with this layout. The converter is 170 lines, easy to audit, and fails loudly.

The freestanding runtime

• Memory: malloc uses AllocatePool with a small header that stores the size (needed by realloc). Code calls the x wrappers (xmalloc, xstrdup, xasprintf…), which stop the shell with rt_fatal on out-of-memory, so callers never check for NULL.
• printf: fmt.c implements vsnprintf with %d %i %u %x %X %o %c %s %p %%, flags, width, precision and the length modifiers hh h l ll z j t. There is no %f.
• Strings: libc.c has the usual mem*/str* functions, strto* and qsort; rt.h has inline ctype functions (ASCII only).

The platform layer (PAL)

src/pal/pal.h is the contract between the portable code and the machine. Everything in it has two implementations: pal_efi.c and pal_host.c.

The contract

pal_efi.c

• Console: output converts UTF-8 to UCS-2 and calls ConOut->OutputString (turning \n into \r\n); input uses SimpleTextInputEx when available (modifier keys) and WaitForEvent with a timer for timeouts. pal_con_ansi() is false: the editor and the menus position the cursor absolutely.
• Volumes: every handle with SimpleFileSystem becomes a volume; they are sorted by device path text so that numbering is stable, and the one whose device matches the loaded image is the boot volume.
• Files: efi_open_path opens the volume root and then the path through EFI_FILE_PROTOCOL. pal_rename uses SetInfo with a new file name, so it only works within one volume (the core falls back to copy + delete).
• Extras for EFI modules (efi_glue.h): the globals gST, gBS, gRT, gImage, device path helpers, efi_volume_handle, efi_file_devpath.

pal_host.c and host.c

The Linux build maps volumes to directories: NESH_FS="dir1;dir2" gives fs0: and fs1: (default: the current directory). The console uses termios raw mode when stdin is a terminal and plain line reading otherwise, which is what the tests use. Other environment variables of the host build:

src/platform/host.c implements the platform hooks with stubs: running .efi files fails, UEFI variables do not exist, only a minimal set of platform commands is registered.

The shell core

src/core turns lines into actions: it keeps the command registry, splits and executes command lines, manages output, the environment, aliases and the prompt.

Commands and the registry

A command is a C function int fn(int argc, char **argv) described by a Cmd entry. Modules keep a static table and register it at start-up with shell_register; the registry is a sorted array of pointers, searched case-insensitively by shell_find_cmd.

typedef int (*CmdFn)(int argc, char **argv);

typedef struct {
    const char *name;
    CmdFn fn;
    const char *usage;   /* "cp [-r] SRC... DST" */
    const char *summary; /* one line */
    const char *help;    /* detailed help, may be NULL */
    int flags;           /* CMD_* */
} Cmd;

#define CMD_KEEP_QUOTES 1 /* arguments keep their double quotes (setvar "text" vs L"text") */
#define CMD_DATA 2        /* supports "-data" (key=value records, see out_data_mode) */

• argv[0] is the command name; the strings belong to the caller.
• The return value is the exit code: RC_OK 0, RC_FAIL 1, RC_USAGE 2, RC_NOTFOUND 127, RC_BREAK 130.
• -data is removed from argv by shell_exec_argv before the call; the command sees out_data_mode(). A command without CMD_DATA never receives it (the core rejects the option), and -sfo is rejected for every command.
• The help texts are shown by help NAME and extracted into the user manual; see The documentation system.

Helpers for commands: cmd_err(cmd, fmt, …) prints cmd: message on the error output and returns RC_FAIL; cmd_perr formats a PAL error for a path; cmd_usage prints the usage line and returns RC_USAGE; getopts(argc, argv, "lr", flags) parses single-letter options that may appear anywhere and returns the index of the first operand.

Splitting and redirection

split_ex in shell.c splits a line at spaces and tabs. Double quotes group words and "" inside quotes is a literal quote; with CMD_KEEP_QUOTES the quotes are kept, because setvar distinguishes "text" from L"text". An unquoted > file or >> file is removed from the words and becomes a file sink around the command. There is no variable expansion and no pipe.

Output sinks

flowchart LR
  W["out_write / out_printf"] --> T{"top of the sink stack"}
  T -- "none" --> C["console (pal_con_write)"]
  T -- "capture" --> B["Sbuf (RUN$)"]
  T -- "file" --> F["PalFile (> file, RUN … TO)"]
  E["err_printf"] --> C

con.c keeps a stack of up to 16 sinks. Normal output goes to the innermost one, so nested RUN$ captures and redirections compose naturally; error messages always go to the console. Colors are only applied when the output is the console (out_is_console). The -data helpers live here too: data_record() starts a record (an empty line before every record but the first), data_field(key, fmt, …) writes one line (newlines in values become spaces), and info_line prints either Label: value or label=value depending on the mode, so a command can describe fields once for both outputs.

UEFI applications started by NESH write to the firmware console directly. When the output is not the console, efi_start_image swaps gST->ConOut for a proxy that forwards text to the current sink; see Running images.

Paths

Inside NESH every path is canonical before it reaches the PAL: fsN:\dir\file, backslashes, no . or ... path_resolve does the conversion: it resolves map aliases (usb:), relative paths against the current directory, / separators, and returns NULL if the volume does not exist. Wildcards are expanded by path_glob (one directory level, sorted, case-insensitive match); commands decide whether to accept them.

Environment variables and aliases

env.c keeps variables in memory; non-volatile ones are also written through the platform hook platform_env_store, which stores them as UEFI variables with the EDK2 shell environment GUID, exactly like the UEFI Shell (so both shells share them). A few names are computed on read (cwd, lasterror, versions). The overlay (env_overlay_push/pop) temporarily hides or replaces variables without touching the firmware; it implements the environment parameter of EFI_SHELL_PROTOCOL.Execute. Aliases (alias.c) work the same way with the shell alias GUID.

Prompt, line editor, history and Ctrl-C

• The REPL (repl() in shell.c) reads a line with lineedit_read, asks the interpreter whether it is BASIC (interp_is_basic_line) and otherwise runs it as a command. An incomplete BASIC block is accumulated until the parser accepts it, with a ... prompt.
• lineedit.c works in two modes: ANSI escape sequences (Linux terminals) or absolute cursor positioning (UEFI consoles, which have no ANSI support). History has 500 entries and is appended to \nesh_history.txt on the boot volume.
• Ctrl+C: con_poll reads pending keys into a type-ahead buffer and sets a break flag when it sees Ctrl-C; on UEFI a key notification registered by the shell protocol module catches it even while an application runs. Long loops call con_break(); the interpreter checks it before every statement.

The BASIC interpreter

NESH BASIC is implemented as a classic pipeline: a lexer produces tokens, a recursive-descent parser builds an abstract syntax tree, and a tree-walking interpreter executes it.

flowchart LR
  SRC["source text"] --> LEX["lexer.c<br/>tokens"]
  LEX --> PAR["parser.c<br/>Program: Block + Procs"]
  PAR --> INT["interp.c<br/>exec_block / eval"]
  INT --> BF["built-ins<br/>bfuncs.c, efi_var.c, efi_boot.c"]
  INT --> SH["shell core<br/>RUN / RUN$"]

Lexer

lex_next produces one Token at a time: numbers (decimal, 0x, 0b, &H, &B, &O, with _ separators; overflow is an error), strings ("" escape, no newline inside), identifiers (a trailing $ is part of the name), keywords (matched case-insensitively from a table), punctuation and newlines. Comments (', REM) and the line continuation _ are skipped here, so the parser never sees them. A UTF-8 BOM and a #! first line are ignored. Errors are sticky: after T_ERROR the lexer stops.

Parser and AST

The parser keeps one token of look-ahead. Expressions follow this precedence (lowest first): OR/XOR → AND → NOT → comparisons → SHL/SHR → + - → * / \ MOD → unary sign → ^ (right-associative) → primary. Depth is limited (64 levels) to protect the stack.

Statements are parsed by parse_block, which loops until it meets a terminator (NEXT, WEND, LOOP, ELSE, ELSEIF, CASE, END IF, END SUB…) and returns which one it found; the caller checks that it is the expected one and reports mismatches with the line where the block was opened. A label is an identifier followed by : alone on its line; labels are recorded per block with the index of the next statement. SUB/FUNCTION definitions are only allowed at the top level and are collected in Program.procs.

When the parse fails because the input ended inside an open block, Program.incomplete is set: the REPL uses it to ask for more lines instead of reporting an error.

Values, strings and variables

A Value is either a 64-bit integer or a pointer to a reference-counted Str (length + bytes, always NUL-terminated). Copying a string value only increments the count (v_copy); v_free releases it. Arithmetic wraps around (it is done on unsigned integers to avoid undefined behavior); division by zero is a run-time error.

Variables live in hash tables (VarTable, FNV-1a hash). There is one global table and, during a procedure call, a Frame with the local table; lookup checks the current frame's locals, then the globals — never the caller's locals. A variable is created on first assignment; reading an unknown variable gives 0 or "". Arrays are the same Var with is_array set and a vector of values.

Execution and control flow

exec_block runs statements in order and returns an Exec code that tells the enclosing constructs what happened:

This explains the GOTO rule: a block that does not contain the label returns X_GOTO to its parent, so a jump can leave loops but never enter them. rt_err records the first error with its line and returns false; every evaluation function propagates false upward, and finish_exec prints file:line: error: message.

Procedures: call_proc evaluates the arguments in the caller's context, type-checks them against the parameter names, creates a Frame, runs the body and converts X_RETURN into the return value. The call depth is limited to 200.

RUN, RUN$ and exit

basic_run_command implements both: with one argument it calls shell_exec_line, with several it builds an argv and calls shell_exec_argv; RUN$ pushes a capture sink first and strips the final newlines. The exit code goes to in->err (the ERR function). After the command it checks two flags set by the exit command:

• shell_exit_requested (exit CODE): the script ends like END CODE; every calling script sees the same flag after its own RUN and ends too, and finally the REPL loop stops.
• shell_script_exit_requested (exit /b CODE): the innermost script clears the flag and ends with CODE, which its caller receives in ERR.

Interactive mode

The REPL keeps one long-lived interpreter, so variables and SUBs defined at the prompt persist; programs that define procedures are kept alive (in->progs) because the procedures point into their AST. Scripts, instead, each get a fresh interpreter from basic_run_file. interp_is_basic_line decides whether a prompt line is BASIC: it is if it starts with a statement keyword (but a lone exit is the shell command), assigns (x =, a(1) =) or calls a SUB defined earlier.

Built-in functions

Built-ins are C functions registered in tables of BFunc:

/* Built-in functions. Argument type letters: 's' string, 'n' number,
 * '?' any, 'a' array (passed unevaluated, see interp_array_arg).
 * The last letter repeats for extra arguments. */
typedef bool (*BFn)(Interp *in, Node *call, Value *args, int nargs, Value *out);
typedef struct {
    const char *name; /* lower case, with '$' for string results */
    int minargs, maxargs;
    const char *types;
    BFn fn;
} BFunc;

call_builtin checks the number of arguments, evaluates and type-checks them according to types, calls the function and frees the arguments. A function returns false only for a run-time error (after rt_err); functions that can fail softly (files, variables) set in->err instead, so scripts test ERR. The result type must match the name: a name ending in $ returns a string. Array arguments ('a') are not evaluated; the function gets the variable with interp_array_arg. Portable functions are in bfuncs.c (basic_core_funcs_init); UEFI ones are registered by efi_var.c and efi_boot.c.

UEFI modules

Everything that needs the firmware lives in src/platform/efi. The modules register their commands and BASIC functions from platform_cmds_init.

Handle numbers

UEFI handles are opaque pointers. efi_drivers.c keeps a list of every handle ever seen (hdb); a handle's number is its position in that list, printed in hex. The list only grows, so a number keeps its meaning for the whole session, even after other handles disappear. efi_handle_index and efi_parse_handle convert in both directions.

Running images

efi_start_image runs applications (load and loadpcirom start drivers with LoadImage/StartImage directly, without shell parameters):

1. Read the file and call LoadImage with its device path (the firmware checks the Secure Boot signature here).
2. Put the command line (UCS-2) into LoadedImage.LoadOptions, as UEFI loaders expect.
3. Install EFI_SHELL_PARAMETERS_PROTOCOL on the image handle: Argc/Argv and the three standard file handles, as ShellLib programs expect.
4. If output is being captured or redirected, replace gST->ConOut with the proxy (and fix the system table CRC); the proxy's OutputString writes into the current sink.
5. StartImage; then restore everything, print the exit data if any, and map the status to an exit code.

EFI_SHELL_PROTOCOL

Programs built with the EDK2 ShellLib locate EFI_SHELL_PROTOCOL and call it for files, environment, current directory, mappings and to run commands. NESH implements all of version 2.2 (efi_shellproto.c).

• Installation: if another shell's protocol exists (NESH started from the UEFI Shell), NESH replaces it with ReinstallProtocolInterface and restores it on exit; otherwise it installs its own on its image handle.
• File handles: a SHELL_FILE_HANDLE is a NeshFile whose first member is an EFI_FILE_PROTOCOL, so programs can also call the file methods directly. Special kinds implement standard input, output (into the current sink), error and NUL.
• Memory ownership: data returned for the caller to free (file info, device paths, help text, file lists) is allocated with AllocatePool; strings returned as CONST stay owned by NESH (small caches keep them alive).
• Execute runs a command line through shell_exec_line, with the environment overlay when the caller passes an environment.

tests/apps/shelltest.c exercises every function of the protocol and is run by make qemu-test.

UEFI variables and boot entries

efi_var_read / efi_var_write wrap GetVariable/SetVariable with UTF-8 names. GUID names (global, security, shell…) are resolved by guid_parse. efi_boot.c parses EFI_LOAD_OPTION structures (attributes, description, device path list, optional data) and never writes a variable directly: changes are collected in a plan.

Plan p = { .dry = a->n, .yes = a->y, .nobackup = a->B };
plan_add(&p, "BootOrder", data, size, xstrdup("new boot order ..."));
int rc = plan_apply(&p); /* show; stop if dry run; confirm; back up; write */
plan_free(&p);

This one function gives every bootmgr subcommand the same dry run, confirmation and backup behavior.

Network

flowchart BT
  SNP["SimpleNetwork (NIC driver)"] --> MNP["MNP"]
  MNP --> IP4["IP4 / IP4Config2"]
  MNP --> IP6["IP6 / IP6Config"]
  IP4 --> UDP4["UDP4"] --> MT4["MTFTP4 → tftp"]
  IP4 --> TCP["TCP"]
  IP6 --> UDP6["UDP6"] --> MT6["MTFTP6 → tftp"]
  IP6 --> TCP
  TCP --> HTTP["HTTP → http"]
  IP4 --> PING4["ICMP → ping"]
  IP6 --> PING6["ICMPv6 → ping6"]

NESH uses the firmware's protocols through their service binding: CreateChild gives a private instance, which is configured, used and destroyed. Configuration commands talk to IP4Config2 and IP6Config. Ping builds ICMP echo packets itself and sends them through a raw IP instance; for ICMPv6 NESH also computes the checksum over the IPv6 pseudo-header. net6_prepare picks the IPv6 source address that can reach the destination (link-local for fe80::, otherwise a global one) and waits up to 10 s for router autoconfiguration. tftp uses MTFTP4 or MTFTP6, which have the same function table and token, so only the configuration differs; http selects the IPv6 access point when the URL host is in brackets.

UEFI compression

src/lib/eficomp.c implements the EFI 1.1 compression format used by option ROMs and firmware volumes, both directions: LZ77 with an 8 KiB window, blocks with three canonical Huffman trees (NT = 19 for code lengths, NC = 510 for literals and match lengths, NP = 14 for position classes), bits written most significant first. The output is verified against the firmware's own decompressor in the QEMU tests.

Secure Boot guard

secure_boot_active() reads the SecureBoot variable. Low-level writes call hw_write_allowed("cmd") first, which prints the standard message and returns false while Secure Boot is active: today mm writes and hexedit -m saves. A new command that writes memory, I/O ports or PCI registers must use it.

Extending NESH

Two complete walkthroughs: a new command and a new BASIC function. The code below is compiled and tested as shown.

Tutorial: adding a command

We add sum, which prints the size and a 16-bit checksum (the BSD sum algorithm) of files, with a -q option and -data support. It only uses the PAL through the core, so it is portable: it goes in src/cmd and works in both builds.

1. Write the command in a new file src/cmd/cmd_sum.c:

/* sum: size and 16-bit checksum of files (tutorial example of the developer manual). */
#include "../core/shell.h"

static int cmd_sum(int argc, char **argv)
{
    bool f[1];
    int first = getopts(argc, argv, "q", f); /* -q: only the checksum */
    if (first < 0)
        return RC_USAGE;
    if (first == argc)
        return cmd_usage("sum");
    int rc = RC_OK;
    for (int i = first; i < argc; i++) {
        char *path = path_resolve(argv[i]);
        char *data;
        size_t len;
        int e = path ? file_read_all(path, &data, &len) : PAL_ENOENT;
        if (e) {
            rc = cmd_perr("sum", argv[i], e);
            free(path);
            continue;
        }
        unsigned s = 0;
        for (size_t k = 0; k < len; k++) {
            s = (s >> 1) | ((s & 1) << 15); /* BSD sum: rotate right, then add */
            s = (s + (uint8_t)data[k]) & 0xFFFF;
        }
        if (out_data_mode()) {
            data_record();
            data_field("path", "%s", path);
            data_field("size", "%zu", len);
            data_field("sum", "%04X", s);
        } else if (f[0]) {
            out_printf("%04X\n", s);
        } else {
            out_printf("%04X %8zu  %s\n", s, len, path);
        }
        free(data);
        free(path);
    }
    return rc;
}

static const Cmd sum_cmds[] = {
    { "sum", cmd_sum, "sum [-q] FILE...", "Show the size and a 16-bit checksum of files",
      "  -q        print only the checksum\n"
      "  sum *.efi              one line per file: checksum, size, path\n"
      "With -data: path, size, sum.\n", CMD_DATA },
};

void cmds_sum_init(void)
{
    shell_register(sum_cmds, ARRAY_SIZE(sum_cmds));
}

Points to notice:

• getopts handles -q anywhere on the line and returns the first operand.
• Paths are made canonical with path_resolve before any file function.
• Errors are reported with cmd_perr/cmd_err and the loop continues with the next file; the exit code remembers the failure.
• -data needs no parsing: the flag CMD_DATA and out_data_mode() are enough.
• The help text follows the rules: lines of at most 78 columns, options first, examples, the -data fields last.

2. Register it. Declare the init function in src/core/shell.h next to the others (void cmds_sum_init(void);), call it from nesh_main in src/core/shell.c after cmds_edit_init(), and add src/cmd/cmd_sum.c to COMMON_SRC in the Makefile. (A UEFI-only command goes in src/platform/efi instead: add its table to one of the modules or register it from platform_cmds_init.)

3. Give it a chapter in the reference: add "sum" to a category of CATEGORIES in tools/gen-docs.py (for example “Text and data”) and run make docs. make test fails until this is done, so a command can never be left out of the manual.

4. Test it. Add a host test, e.g. tests/host/sum.nsb, and generate its expected output:

WRITEFILE "a.txt", "hello" + CHR$(10)
RUN "sum a.txt"
RUN "sum -q a.txt"
PRINT FIELD$(RUN$("sum -data a.txt"), "size")
RUN "sum missing.txt"
PRINT "ERR="; ERR

$ make build/nesh-host
$ UPDATE=1 tests/run-host-tests.sh build/nesh-host   # writes tests/host/sum.out
$ cat tests/host/sum.out                              # check it by hand once
$ make test

The expected output is 9073 6 fs0:\a.txt, then 9073, then 6, then the error message and ERR=1 (0x9073 = 36979, the same value the Unix sum command prints). If the command uses firmware services, also add checks to tests/efi/run.nsb and run make qemu-test.

Tutorial: adding a BASIC function

We add REVERSE$(s$), which reverses a string character by character (not byte by byte: UTF-8 characters must stay intact). Portable functions go in src/basic/bfuncs.c:

/* REVERSE$(s$): the characters of s$ in reverse order. */
FN(f_reverse)
{
    UNUSED;
    const Str *s = a[0].s;
    Sbuf b;
    sb_init(&b);
    for (size_t i = s->len; i > 0;) {
        size_t start = i - 1;
        while (start > 0 && ((uint8_t)s->s[start] & 0xC0) == 0x80) /* UTF-8 continuation byte */
            start--;
        sb_add(&b, s->s + start, i - start);
        i = start;
    }
    *out = v_str(str_take_sb(&b));
    return true;
}

and one line in the core_funcs table (name in lower case with the $, one argument, type letter s):

    { "reverse$", 1, 1, "s", f_reverse },

That is all: the parser needs no change, because any name(args) is resolved at run time. PRINT REVERSE$("NESH è bello") prints olleb è HSEN. Then document the function in the function reference of the user manual (a <dt id="fn-…"> entry): make test fails for a built-in function without an entry.

Testing

Three levels of tests, all automatic: host tests for the language and portable commands, documentation checks, and QEMU tests for everything that needs firmware.

flowchart LR
  subgraph T1["make test (seconds)"]
    H["tests/host/*.nsb<br/>vs *.out"]
    G["gen-docs.py --check"]
    X["check-doc-examples.py"]
  end
  subgraph T2["make qemu-test (~1 min)"]
    Q["tests/efi/run.nsb<br/>in OVMF"]
    SA["shelltest.efi<br/>EDK2 apps"]
  end
  subgraph T3["make qemu-nettest"]
    N["tests/efi/net.nsb<br/>IPv4 + IPv6"]
  end

Host tests

tests/run-host-tests.sh copies each tests/host/NAME.nsb into an empty temporary directory, runs it with NESH_FS=. and the arguments arg1 "arg two", and compares stdout+stderr and the exit code with NAME.out. UPDATE=1 rewrites the .out files — review the diff before committing. The host build uses AddressSanitizer and UBSan, so memory errors and undefined behavior make tests fail loudly.

QEMU tests

tools/run-qemu.sh --test builds a virtual FAT disk (build/esp, read-only, becomes fs0:) with nesh.efi as \EFI\BOOT\BOOTX64.EFI, the tests, the examples and the fixtures, plus a writable FAT image (build/work.img, fs1:). OVMF boots NESH, which runs startup.nsb → tests\run.nsb and shuts the machine down. The serial output is saved in build/qemu-test.log. The run passes only if the log contains the end marker, the line failures:0 and no line starting with FAIL.

Inside the tests, a small SUB does the bookkeeping:

fails = 0
SUB check(name$, ok)
  IF ok THEN
    PRINT "PASS "; name$
  ELSE
    PRINT "FAIL "; name$
    fails = fails + 1
  END IF
END SUB
...
check "map lists fs0", INSTR(RUN$("map"), "fs0") > 0

--nettest adds a virtio network card with QEMU user networking (DHCP on 10.0.2.0/24, router advertisements for fec0::/64, a TFTP server) and an HTTP server on the host, and loads the firmware network drivers from tests/fixtures/netdrv.

Continuous integration and releases

.github/workflows/ci.yml runs on GitHub Actions for every push to main and every pull request: it installs QEMU, OVMF, iPXE's option ROMs and dosfstools on Ubuntu, enables KVM, and runs make, make test, make qemu-test and make qemu-nettest; the QEMU logs and nesh.efi are kept as artifacts.

Releases are made by pushing a version tag:

$ git tag -a v0.2.0 -m "NESH 0.2.0"
$ git push origin v0.2.0

The same workflow builds and tests the tagged source, then publishes a GitHub Release with nesh.efi and SHA256SUMS. Update NESH_VERSION in src/core/shell.h (and the version shown in the manuals) before tagging.

Secure Boot tests

make qemu-sbtest checks the Secure Boot behavior end to end. It creates test keys (PK, KEK, db) with openssl, enrolls them into a copy of the OVMF variable store with virt-fw-vars, signs nesh.efi with sbsign and boots the OVMF build that enforces Secure Boot (OVMF_CODE_4M.secboot.fd; OVMF_SECBOOT= and OVMF_VARS= override the paths). Two runs:

1. the signed shell runs tests/efi/secureboot.nsb, which checks that NESH reports Secure Boot as active, that reading hardware still works, that low-level writes are refused, that clock and serial settings are still allowed, that unsigned applications and drivers are refused with a clear message, that ordinary UEFI variables can still be written and that authenticated ones (the keys) cannot;
2. the unsigned image must not start at all (the firmware answers Access Denied).

The keys are generated once in build/secureboot/. Without sbsign, virt-fw-vars or a Secure Boot OVMF build the target prints why it is skipped and succeeds.

Fixtures

The documentation system

• Command reference: tools/gen-docs.py reads every static const Cmd …[] table in the sources (a small C tokenizer joins adjacent string literals and expands NESH_VERSION/SCRIPT_EXT) and rewrites the part of docs/user-manual.html between the BEGIN GENERATED/END GENERATED markers. --check fails when the manual is out of date, when a command has no category, when a help line is longer than 78 columns, or when a BASIC function has no entry in the function reference.
• Line counts: the same tool counts the files of the source map and the totals of the area table and writes them into docs/developer-manual.html, so the numbers cannot drift away from the tree. --check fails when they are stale, when the map lists a file that does not exist and when a source file has no row at all: adding a file to src, include, tools or tests/apps therefore means adding a line to the map.
• Examples: tools/check-doc-examples.py takes every <pre data-example="NAME"> of the user manual, writes it as NAME.nsb, checks the syntax of all of them (nesh -k), then runs the ones that have an output block (data-example-output="NAME") in page order and compares the output. Examples marked data-platform="efi" are only syntax-checked.
• Look: chosen by comparing ten proposals built by tools/make-previews.py into docs-preview/ (not published, not in the repository). Run it again to try another design: each proposal is a layout plus a palette, applied to the same sample content.
• Pages: plain HTML with assets/docs.css and assets/docs.js (table of contents, numbering, search, index, syntax colors) and a local copy of Mermaid for the diagrams, so the manuals work offline. In a Mermaid block write line breaks as <br/>: a literal tag would be eaten by the HTML parser.

Conventions

Code style

• C11 with GNU extensions, 4-space indentation, braces on the same line, snake_case; file-local functions and data are static.
• Comments explain why, in English, and are short. Every file starts with a comment saying what it contains.
• Text inside NESH is UTF-8. Conversion to and from UCS-2 happens only at the firmware boundary (utf8_to_ucs2, ucs2_to_utf8).
• Functions that return allocated memory say so in their comment; the caller frees it. Allocation never fails (x* wrappers).
• Command options: the NESH form and the UEFI Shell form are both accepted when they do not conflict; UEFI Shell options that make no sense in NESH (such as -b) are accepted and ignored, never rejected.
• No code, headers or libraries from EDK2. Protocol and structure definitions are written from the UEFI specification.
• Keep the build free of warnings and the tests green; add a test with every fix.

Help text rules

• summary: one line, at most 78 columns, starting with a verb (“List…”, “Show…”).
• help: option lines indented by two spaces with aligned descriptions, then short notes, then 1–3 examples; every line at most 78 columns (checked by make test).
• Document only what the code does. Options accepted for compatibility and ignored are said to be ignored.
• With CMD_DATA, the last line lists the -data fields.

Recording decisions

Design decisions — what was chosen, the alternatives and why — are recorded in docs/decisions-and-history.md. Read it before changing a behavior that looks arbitrary: there is usually a reason.

Debugging

make qemu gives an interactive NESH on the serial console of your terminal; Ctrl+A X leaves QEMU. Debug output can be printed with err_printf, which always reaches the console even while output is captured.

Source file map

Glossary

Boot services

Firmware functions available until the OS takes over (gBS): memory, events, protocols, images.

Device path

A chain of nodes describing where a device or file is, e.g. PciRoot(0x0)/Pci(0x2,0x0)/HD(1,…).

Driver binding

The protocol through which a driver is connected to controllers.

ESP

EFI System Partition: the FAT partition with boot loaders.

GUID

128-bit identifier of protocols and variable namespaces.

Handle

An object in the firmware database carrying protocols.

Load option

The content of a Boot####/Driver#### variable.

OVMF

The UEFI firmware for virtual machines built from EDK2; used by the tests.

PE32+

The executable format of UEFI images (the 64-bit Windows format).

Protocol

A table of functions (and data) identified by a GUID and installed on a handle.

Runtime services

Firmware functions still available to the OS (gRT): variables, time, reset.

Service binding

A protocol that creates private child instances of a network protocol.

ShellLib

The EDK2 library that UEFI Shell applications use; it relies on EFI_SHELL_PROTOCOL.

TPL

Task priority level: UEFI's simple interrupt-priority mechanism for events.

UCS-2

16-bit character encoding of UEFI strings (CHAR16).

Index