Repository navigation
libc: POSIX threads, semaphores, regex, glob, nftw, syslog - #167
Merged
Merged
Conversation
…tory once The driver appended <binary's directory>/include as a system directory ahead of the defaults paths.c computes -- a line from before paths.c existed. That directory holds the compiler's declarations-only <string.h>, which paths.c deliberately puts LAST so it cannot shadow the C library's; added early, it did, and every build-tree compile without -I lost the POSIX half of <string.h> (strtok_r, strlcpy, ...). Run from $PATH, argv[0] has no '/', and the directory was ./include -- wherever the user happened to be. It is now added only when paths.c found no support files, which is what it was for. With it gone, a directory named both by -I and by the defaults was searched twice in a row, and #include_next from the C library's <limits.h> found that same file again, include guard set, so INT_MAX was never defined (the libc build itself broke). GCC searches such a directory once, where -I put it, as a system directory; so does EmbCC now, by plat_same_dir: device and inode where the host has them (macOS, Linux), spelling elsewhere -- EmbLinkOS's newlib stat has no inodes. tests/golden/install.sh compiles <string.h> + <limits.h> with and without -I lib/libc/include; the first case fails on the old driver and the second on one without the dedupe. build.ebm regenerated for platform_posix.c's <sys/stat.h>.
The first of the POSIX layer EmbLinkOS needs to leave newlib behind --
the part that asks the operating system nothing new.
<string.h> strtok_r, strerror_r (XSI's), strsep, strlcpy, strlcat,
memmem; strcasestr, memrchr, strchrnul, explicit_bzero
<stdlib.h> setenv, unsetenv (declared before, never defined), putenv,
clearenv; the mkstemp family, mkdtemp, mktemp; realpath;
posix_memalign, memalign, reallocarray; qsort_r; getsubopt;
random/srandom/initstate/setstate, the *rand48 family and
rand_r, each giving glibc's sequence for a seed
modules <libgen.h>, <fnmatch.h>, <getopt.h> (getopt_long, and
permuting as glibc's and newlib's do), <search.h> (AVL
tsearch, a growing hsearch, the _r forms)
<features.h> decides what a C header declares beyond ISO C: everything
unless the program asked for ISO C alone, and GNU's additions hidden
too when it names a POSIX version and nothing more, as glibc does.
The environment owns what it allocates: the startup vector is copied on
the first change, setenv's strings are freed when replaced, putenv's
are the caller's and never are. realpath resolves a component at a time,
so ".." after a symlink leaves the link's target. The entropy behind
temporary names is one function now (__tmp_bits), which stdio's tmpnam
will share.
tests/exec/posix-strings.c holds the values glibc 2.36 printed for the
same calls, run against both on a Linux kernel; where they differ the
test says why (tree shape, a growing hash table). Its tree checks catch
an AVL delete that skips rebalancing.
<stdio.h> gains fdopen, fileno, fmemopen, open_memstream, getline, getdelim, dprintf, asprintf and their v forms, fseeko, ftello, ctermid, flockfile/ftrylockfile/funlockfile and the _unlocked functions, and GNU's fopencookie, setbuffer and setlinebuf. <err.h> is new; its messages begin with program_invocation_short_name, which Linux's startup now sets from argv[0]. What it took inside: a FILE may be something other than a descriptor -- four functions and a cookie, laid out as glibc's cookie_io_functions_t -- and the seven places stdio reached the backend go through one dispatcher each; the stream lock NESTS. flockfile(stdout) and then printf takes it twice, and the old lock, which is not recursive, deadlocked -- and on a target with no futex spun forever, every single-threaded program; freopen reopens the stream it is given. It closed it and called fopen, which handed back a different FILE, so freopen(.., stdout) left stdout closed; past FOPEN_MAX, streams come in chunks from the heap that are never freed, so the flush at exit cannot walk into a closed one. fmemopen follows glibc since 2.22 (where a NUL goes); open_memstream's length moves only when a write passes it, which is POSIX's rule and not glibc 2.36's. tests/exec/posix-stdio.c and posix-files.c (mkstemp, realpath through symlinks, freopen of stdout) run on our libc and, posix-stdio at least, on glibc, on a Linux kernel; linux-libc-exec.sh runs both, and posix-strings, with no -I.
EmbLD wrote every image it linked with the mode a plain file gets, 0644 under the usual umask -- so `embcc hello.c -o hello` on a Linux host made a program that could not be run, and a test that installed one in a guest as /bin/sh got EACCES from the kernel's exec. It now gets 0777 less the umask, as GNU ld gives its output: ELF executables, script-linked images and EMBX alike, through plat_make_executable (chmod where the host has modes, nothing in the ISO host layer).
os/backend.h gains POSIX's half: about a hundred primitives, each
shaped like the call it serves (dup, pipe, fcntl, fstat, the *at forms,
fork/execve/spawn/waitpid, sigaction and the masks, clocks, mmap, ppoll,
sockets, termios, ptys, uname and sysconf's questions), with Linux's
numbers for everything that needs one. A backend that cannot do one
still defines it, failing with ENOSYS: os/absent.inc is the list,
guarded per primitive, which the bare-metal and posixlike backends
include whole.
os/linux/posix.c implements every one as syscalls on both 64-bit ABIs,
the numbers read out of the kernel's own uapi headers: open's flags
translated where aarch64 moves them, the kernel's shorter termios and
utsname converted, a signal restorer placed as bytes (checked against
both assemblers), posix_spawn as fork + actions + execve with the
child's error returned over a close-on-exec pipe.
Above them, src/posix: <unistd.h>, <fcntl.h>, <sys/stat.h>,
<dirent.h> (`.` and `..` returned, positions kept), the *at functions,
the exec family with PATH search and the script rule, <sys/wait.h>,
<spawn.h>, system and popen on posix_spawn (so they work where fork
does not), <sys/mman.h>, poll and select, sockets, inet_pton/ntop,
getaddrinfo over the hosts file, the OS's resolver or a DNS client
(resolv.conf, search domains, TCP for truncated answers), termios and
ptys, the user and group files, sysconf and the rest. Every public
POSIX name is weak -- none is reserved to the implementation, and a
program's own `read` or `getline` must win. sys/types.h takes hosted
widths (64-bit ino_t and dev_t, 32-bit uids).
Signals are two models: the kernel's where it delivers (Linux) and
lib/libc's own elsewhere -- a handler table, a mask and a pending set,
so a blocked raise is delivered on unblock -- and abort now raises
SIGABRT first, ending the program even if a handler returns.
tests/exec/posix-{fd,process,signals,net}.c hold what glibc 2.36 gave
for the same calls on the same kernel (they were run against both);
posix-process is its own /bin/sh and child. linux-libc-exec.sh runs
all 19 of its programs on x86-64 and aarch64 Linux: 76 runs exit 42.
The EmbLinkOS backend now defines every primitive of os/backend.h. It
carries over what EmbLinkOS's newlib adapter (user/lib/syscalls.c,
spawn_compat.inc) worked out POSIX means on that kernel, under its
three honest categories -- REAL, VACUOUS (true whatever is done: a
mask where nothing is delivered), ABSENT (ENOSYS, never pretended):
paths normalized here ("..", ".", the 256-byte limit), the
working directory this library's, a child's named in its
spawn or by PWD
files O_CLOEXEC translated (the kernel's is 0x100000),
O_DIRECTORY and O_NOFOLLOW honoured by looking, rename the
kernel's atomic one -- it was unlink-then-rename, which a
crash between the two turned into a lost destination
directories a snapshot that grows until the kernel's silent
truncation is ruled out -- it was a fixed 512 entries
descriptors dup, dup2, pipes, close-on-exec and the status flags;
record locks ENOSYS
processes posix_spawn on the native actions (descriptor snapshot,
cwd, kept status), attributes refused; waitpid on owned
child handles, a cancelled wait EINTR; fork and exec
ENOSYS
time the RTC in seconds, uptime in milliseconds, a thread's CPU
time, each at the precision it has
memory mmap's refusals where the kernel cannot honour a request
network IPv4 sockets and the kernel's resolver; poll on fd_poll
terminals termios and the window size through terminal control
tests/golden/libc-backends.sh reads the primitives out of os/backend.h
and requires every backend -- bare metal (Cortex-M and AVR), posixlike,
Linux on both ABIs, EmbLinkOS -- to define all 139; it fails naming the
one a backend left out.
…efuse time: - tz.c reads TZ as POSIX writes it (quoted names, Mm.w.d / Jn / n rules, times past 24h or negative), and zone names as TZif files (v1-3, with the footer rule) under TZDIR or /usr/share/zoneinfo; no TZ is /etc/localtime, an unusable one is UTC. tzset sets tzname/timezone/daylight. - struct tm carries tm_gmtoff and tm_zone; strftime's %z and %Z read them. - mktime reads tm_isdst as glibc does on the mornings the clocks change. - strptime: the "C" locale's conversions and glibc's readings -- digits stop where one more could not fit, %I joins %p, %C with %y, and the fields a date implies (wday, yday, and a date from %U/%W + weekday). Every value refereed against glibc 2.36 on Linux; tests/exec/posix-time.c holds them and runs in linux-libc-exec on x86-64 and AArch64. spawn: - posix_spawn_file_actions_t and posix_spawnattr_t are handles: init allocates, destroy frees, NULL is EINVAL. - __os_spawn_flags names the attribute flags a backend honours; setflags refuses the rest with ENOTSUP instead of a spawn that ignores them. - posix_spawnp: the system's directories without PATH, and a file the kernel cannot run (ENOEXEC) is run by /bin/sh. EmbLinkOS backend: embk_empty_env for its crt0; libc-emblinkos also makes an empty libm.a, so -lm finds no other library's. libc.sh: the harness's kill (newlib's floor, always EINVAL) is left out, so posix-signals reaches the library's own -- all 366 programs pass.
What EmbLinkOS's own tests asked for, building every program it has against this library (LIBC=emblibc in EmbLinkOS): - kill of another process is ENOSYS where the kernel cannot signal one (ESRCH claimed there was none; mrsh's job control hung on it). - spawn: POSIX's EBADF for a descriptor at or past OPEN_MAX from the add call itself, and __os_spawn_check, a new primitive, lets a backend refuse at that moment what its spawn can never do (EmbLinkOS: more than 61 actions, a path past 255 bytes, fchdir, an open flag it has no bit for). - EmbLinkOS backend: getrlimit(RLIMIT_NOFILE) is the kernel's 64, and fcntl(F_SETFL) with a flag it cannot set is EINVAL, as that OS says. - getpass, with echo off; <malloc.h> (malloc_usable_size, mallinfo2, malloc_stats, valloc, pvalloc -- QuickJS sizes its heap with the first); <string.h> brings <strings.h> and <stdlib.h> <alloca.h>, as glibc's do; ARG_MAX gives way to a build's own. C++ on EmbLinkOS: make libcxx-emblinkos, and libc-emblinkos now also makes librt.a with lib/rt/unwind.c (-D__EMBCC_RT_UNWIND) -- its programs start with -nostartfiles, so nothing registers tables for libgcc's. Everything there is built with --unwind-tables: for x86_64-elf C code has none by default, and the unwinder's own frames need them or every throw ends where it starts. EMBLINKOS_ARCH=aarch64 builds the other set. Fixed on the way: - <filesystem> kept its own copy of __os_fileinfo, shorter than the backend's since the POSIX fields: every status() wrote past it (a hang). libc-backends.sh now compares libcxx's copies of the contract with it, structs and all 45 prototypes; it fails on the old copy. - 16-bit int: tz.c's 1 << 18 buffer and an unsigned-short compare that made week 0 a huge day number on AVR; malloc's 64 KB chunk constant. Tests: malloc-gnu.c (refereed against glibc), posix-process's EBADF; libc.sh 367, linux-libc-exec 84 runs, libcxx-std's struct tm is 56 bytes now, glibc's.
- newlib's limits.h guards itself with _LIBC_LIMITS_H_, the word this library's used both as its guard and to tell the compiler's half it was there. With newlib's headers first (tools/build-libstdcxx.sh puts them there) its #include_next reached ours, ours was already "done", and the compiler's half -- INT_MAX -- was never read: libsupc++ and the libstdc++ tests failed to compile. Ours has its own guard now, sets the shared word only when it is the first C library's half, and always passes the chain on. - posix-process read argc/argv before knowing it ran where a kernel passes them: on Xtensa and the ESP32-C3, whose startup promises no argc, argv[1] was a wild load. The /bin/sh behaviour is under EMBCC_TEST_HAS_FS with the rest of what needs a filesystem. Full matrix: 840/860; these were the two real failures. The others -- timeouts and one killed run, all on the busy machine -- pass alone, as do these two: libc-limits, libm-long-double, rt, sroa, stack-args, frameless, cxx-libstdcxx, cxx-libsupcxx, xtensa-exec, embsim-esp32c3, install, libc-system, big-functions, linux-libc-exec, exec-boards.
The bare-metal and posix-like backends implement __os_write by calling write, __os_sbrk by calling sbrk: those names are the program's or the board's (weak defaults it replaces), newlib-style. The POSIX layer defined them too, weakly, calling the primitives back, so whichever an image linked first won -- and when proc.o came first on Xtensa and MIPS, sbrk called __os_sbrk called sbrk until the stack ran out. open, close, read, write, lseek, isatty, _exit and sbrk move to os/posix-names.c, built for Linux and EmbLinkOS only.
- <pthread.h>: threads (create, join, detach, exit, once, atfork, GNU's tryjoin/timedjoin/setname/getname), attributes, the three kinds of mutex with timedlock and clocklock, condition variables with a clock and clockwait, read-write locks (readers first, as glibc's default), barriers, spin locks, keys with destructor rounds. What cannot be honoured is refused: process-shared objects, robust and priority mutexes ENOTSUP, cancellation ENOTSUP (no cancellation points), a policy other than SCHED_OTHER EPERM. - <semaphore.h>: unnamed semaphores; sem_open ENOSYS, pshared ENOTSUP. - One thread table (src/threads/pthread.c) serves both spellings, keys live in each thread's record (no TLS needed beneath them), and pthread_exit longjmps to its thread's start, where destructors run. fork runs atfork's handlers through a weak reference. C11's layer is rewritten on it, which fixes three bugs it had: - mtx_lock could sleep holding the lock: a lock freed between its try and its exchange was taken by the exchange and then slept on. The three-state loop is the standard one now. - mtx_timedlock and cnd_timedwait read C11's absolute TIME_UTC deadline as a duration -- decades, for any real deadline. - thrd_exit called exit(), ending the program from any thread. tss_* work now (they always failed). timespec_get reads the backend's real-time clock to the nanosecond, not __os_time's whole seconds, so "now plus 30 ms" is not already past. Linux threads get glibc's 8 MiB stacks, mapped on demand. Where there is no futex (one thread), a timed condition wait waits out its deadline instead of returning at once. tests/exec/posix-threads.c: glibc 2.36 and this library both exit 42 on x86-64 and AArch64 Linux; on the boards (no threads, maybe no clock) it checks what one thread can show.
- <regex.h>: POSIX basic and extended syntax with glibc's numbers and messages, GNU's \+ \? \| \< \> \b \B \w \W \s \S, back-references, REG_STARTEND; POSIX's leftmost-longest match by backtracking, a single-character repeat as a loop (no stack per character), and REG_ESPACE instead of a hang past 20 million steps at one position. tests/exec/posix-regex.c: 88 cases whose expected answers are glibc's own output for the same calls. - <glob.h>: every flag glibc has, braces and tildes included; <ftw.h>: nftw with FTW_ACTIONRETVAL, and ftw. - <syslog.h>: RFC 3164 lines to /dev/log. With no logger there, a line is not dropped as glibc drops it: /dev/console under LOG_CONS, else standard error -- EmbLinkOS has no syslog daemon and says a capability must never silently vanish. - <langinfo.h>, <nl_types.h>: the "C" locale's items, CODESET "UTF-8"; no message catalogs, catgets returns the program's own text. posix-glob.c (glob, nftw, ftw, syslog's LOG_PERROR line) and posix-regex.c run in linux-libc-exec with glibc's answers; on the boards they check what needs no filesystem. The manual lists it all.
glibc 2.34 folded those libraries into libc and keeps them as empty archives, so Makefiles still name them; EmbCC's libc has POSIX threads, the clocks and openpty in libc.a. The driver takes the names as its own, as it takes -lm, instead of "cannot find libpthread.a". A program that calls what the library lacks (dlopen) still fails the link, naming it. tests/golden/linux.sh links one through the driver and runs it.
…ak symbol A weak reference left undefined has no PC-relative address in a position-independent program, and EmbCC's x86-64 code addresses everything PC-relative: every EmbLinkOS app (linked PIE) that pulled in fork failed to link on __pthread_atfork_run. The thread library sets __atfork_hook when it is first used, or on pthread_atfork, so a program that never names a pthread function still does not link it -- and a forked child of a threaded program always learns it has one thread. On EmbLinkOS (LIBC=emblibc, its `test pthreads` running this tree's tests/exec/posix-threads.c): exit 42, beside cxxlib, posix, posix-shell, spawn/process-compat, fd-sharing, terminal, tcc, embbuild, env, tz.
…'s does 604db0f moved read, write, open, close and _exit there, out of the portable half this golden builds; without it the archive it checks for closure needs them from nowhere.
teo1747
marked this pull request as ready for review
October 30, 2026 07:07
This was referenced Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #166. Until #166 is merged, the diff here shows it too. The commits after
604db0f5are this PR (plus one golden fix also on #166).What it adds
POSIX threads and semaphores.
<pthread.h>covers threads, attributes, the three kinds of mutex (withtimedlock/clocklock), condition variables with a clock, read-write locks, barriers, spin locks and keys.<semaphore.h>has unnamed semaphores. What can't be honoured is refused with an error, never accepted and ignored: process-shared and robust objects, cancellation, real-time policies.C11 threads on the same core, which fixes three bugs it had:
mtx_lockcould sleep while holding the lock.mtx_timedlockandcnd_timedwaitread the absolute deadline as a duration.thrd_exitended the whole program.tss_*work now; before, they always failed.timespec_gethas nanoseconds, as glibc's does.<regex.h>: basic and extended syntax with POSIX's leftmost-longest match. All 88 test cases give glibc's answers.<glob.h>,<ftw.h>,<syslog.h>,<langinfo.h>,<nl_types.h>.The driver accepts
-lpthread -ldl -lrt -lutil, as glibc 2.34+ does, since these functions live in libc.Tested
posix-threads,posix-globandposix-regexexit 42 under glibc 2.36 and under this library on x86-64 and AArch64 Linux.linux-libc-exec,linux(both architectures),libc,libc-backends,threadsafe,libcxx-std, and the boards (exec-boards,xtensa-exec,mips-exec,embsim-riscv,embsim-avr) pass.LIBC=emblibc), itstest pthreadsrunsposix-threads.cwith real kernel threads: exit 42. Getting there found that a weak undefined symbol cannot link into a PIE built by EmbCC, soforknow reachespthread_atfork's handlers through a hook pointer.libc-emblinkos.sh's own archive lackedos/posix-names.c. That fix is also pushed to libc: POSIX, EmbLinkOS's userland on it, and C++ there by EmbCC #166. The rest came from the fresh worktree, which lackedrt-embedded, or from load.EMBCC_VERIFY=1, clean.