Syscall on FreeBSD amd64 with GNU C Inline Assembly

You will need a copy of the FreeBSD source code (https://git.FreeBSD.org/src.git) to understand this article.

After looking around on the internet, it seems to me that nobody really knows how to make syscalls. The ones that do make it from C. If you are writing your own code, I would recommend using syscall(2) from libc - it takes care of the register juggling for you. However, I hate libc with its errno and unnecessary FILE* abstractions, so I tried doing this myself.

One more thing, the manual page syscall(2) contains calling conventions of other architectures. Take a look if you need to know that.

Here is my attempt in C + Inline Assembly. I’m using FreeBSD 14.0, and the code may not apply to other versions of FreeBSD.

If how to use syscall is undocumented enough, GNU C’s inline assembly says that the constraint "a" is machine-dependent and doesn’t explain further. It turns out that Constraints for Particular Machines is documented in another page.

Anyway, here’s how you do syscalls on FreeBSD+amd64.

• The syscall number is put in rax.
• The arguments are put in rdi, rsi, rdx, r10, r8, r9 (you only use the first ones that you need)
• Then you use the instruction syscall
• The return value will be in rax. If an error occured, the carry flag will be set, and rax will contain the positive error number.
• rcx will be clobbered.

This is like the Sys V x86_64 ABI, but not quite.

About r11, I don’t think it will be clobbered. Neither lib/libc/amd64/SYS.h or sys/amd64/amd64/trap.c says anything about this. There has been rumors online about r11 being clobbered; unless I see this in practice, I’ll treat the rumors as unfounded.

To specify r10, r8, r9, you need yet another syntax: register T var_name asm ("eax");, as they are no letter constraits for them in GCC (stated by this StackOverflow answer)).

Musings

The FreeBSD Developers’ Handbook touts the superiority of passing syscall arguments on the stack. This is on the x86 architecture.

This is also why the UNIX® convention of passing parameters to system calls on the stack is superior to the Microsoft convention of passing them in the registers: We can keep the registers for our own use.

Given that amd64, rv64 and aarch64 all use register-based argument passing, this statement certainly did not age well.

This StackOverflow answer omits error handling.

Are R8-R10 actually clobbered after syscall? I don’t know. In my example, lldb says no. The FreeBSD source code lib/libc/amd64/** doesn’t mention that either. This is all very confusing to work with.

Hare’s syscall function is really interesting, and I’m not sure if it’s correct or not. It depends on the normal return values of syscalls not being in the range of (-4096, -1].

fn wrap_return(r: u64) (errno | u64) = {
	if (r > -4096: u64) {
		return (-(r: i64)): errno;
	};
	return r;
};

.section .text
error:
        neg %rax
        ret

.section .text.rt.syscall6
.global rt.syscall6
rt.syscall6:
	movq %rdi, %rax
	movq %rsi, %rdimovq %rdi, %rax
	movq %rsi, %rdi
	movq %rdx, %rsi
	movq %rcx, %rdx
	movq %r8, %r10
	movq %r9, %r8
	movq 8(%rsp), %r9
	movq %rdx, %rsi
	movq %rcx, %rdx
	movq %r8, %r10
	movq %r9, %r8
	movq 8(%rsp), %r9
	syscall
	jc error
	ret

(I fixed the weird register ordering from the original source code.)

export fn mmap(
	addr: nullable *opaque,
	length: size,
	prot: uint,
	flags: uint,
	fd: int,
	offs: size
) (errno | *opaque) = {
	return wrap_return(syscall6(SYS_mmap, addr: uintptr: u64,
		length: u64, prot: u64, flags: u64,
		fd: u64, offs: u64))?: uintptr: *opaque;
};

So, if the memory address returned from SYS_mmap is greater than 0xffffffffffffffff, it gets treated as an error. I tried to do that, and it doesn’t seem like it is possible to map memory there. I guess the number 4096 comes from the memory page size, and that’s what makes this work?