The usual question is: where is the configuration wrong?

That question assumes configuration is one thing in one place. In production it is usually a piece of state copied through several systems, transformed along the way, and activated by more than one process. The value in the source of truth is only the first copy.

A more useful question is: how far did this revision get?

Configuration is a distributed system. It has authorities, replicas, propagation delays, schemas, consumers, version skew, partial failures, and rarely a transaction covering the whole path. Treating it as a map of keys and values hides the part that causes most operational failures: delivery.

The source of truth is only the source

Consider a common path for an application credential:

flowchart TB
    Authority["External authority"] -->|fetch| Sync["Synchronization controller"]
    Sync -->|materialize| Secret["Kubernetes Secret"]
    Template["Pod template"] -.->|references| Secret
    Secret -->|resolve when container starts| Env["Container environment"]
    Secret -->|project through kubelet| File["Mounted file"]
    Env -->|read at startup| Process["Workload process"]
    File -->|reread or reload| Process
    Process -->|validate and activate| Subsystem["Configured subsystem"]

    classDef authority fill:#fff3d6,stroke:#d1a11f,color:#4f3200
    classDef control fill:#efe7ff,stroke:#6a3fd4,color:#20113a
    classDef state fill:#e8f1ff,stroke:#2f5da8,color:#102a43
    classDef runtime fill:#d8f3ef,stroke:#1b8c7a,color:#0f3c36
    class Authority authority
    class Sync,Template control
    class Secret state
    class Env,File,Process,Subsystem runtime

Two delivery paths. The pod template references the Secret but does not contain its value. Environment delivery happens when a container starts; file delivery follows kubelet projection and still requires the process to reread or reload it.

Each edge is a contract.

The controller needs the correct source path, permissions, refresh policy, target name, key mapping, and transformation. The workload manifest needs to reference the correct Secret and key. A running process cannot receive a changed environment variable; a controlled Pod rollout is the normal way to create processes with the new environment. The process must parse the value and make it available to the subsystem that uses it. Every relevant workload needs a complete path of its own.

None of those steps is implied by the previous one. A value can exist in the secret manager without being selected by the synchronization resource. A Kubernetes Secret can be current while existing Pods retain old environment variables. A mounted file can change while an application that read it once at startup continues using the previous value. A web Deployment can be wired correctly while a worker Deployment is forgotten.

Calling the secret manager the source of truth is still useful: it identifies the authority allowed to declare the desired value. It does not prove the state of any running consumer.

Four states and an evidence plane

It helps to give each stage a name. A production configuration change moves through four lifecycle states, while an evidence plane makes every transition verifiable.

Declared. An owner has recorded a desired revision in the authoritative system. This might be a Git commit, a secret-manager version, a configuration-service record, or an API mutation.

Materialized. The desired revision has been rendered into the system-specific object from which a workload can receive it: a Kubernetes Secret or ConfigMap, a generated file, a service response, or a release artifact.

Delivered. The workload has access to that revision. For a container this may mean a new environment, a projected file, or a successful runtime fetch.

Activated. The running process has parsed, validated, and begun using the revision. Delivery and activation are different when applications cache configuration, reload asynchronously, reject invalid updates, or contain several independently configured subsystems.

Evidence plane. For every intended consumer, an operator can determine which revision reached each lifecycle state without exposing its value. Evidence is not another copy of configuration; it is attached to every transition.

A successful declaration is therefore only the start of a change, not evidence that it reached runtime.

This vocabulary also narrows incidents quickly. “The Secret is correct” establishes materialization. It says nothing about delivery or activation. “The rollout finished” establishes that Pods were replaced. It does not prove the process accepted the intended revision. A healthy endpoint may prove the process can serve traffic, but unless health includes the relevant configuration invariant, it may not prove activation either.

Presence is not propagation

Kubernetes makes the distinction concrete. A ConfigMap projected as a normal volume is eventually updated by the kubelet, subject to its synchronization and cache propagation delays. The application uses the new value only if it rereads or reloads the file; one that reads only at startup continues using its previous in-memory state. A subPath mount is an important exception because it does not receive ConfigMap updates. A running process never observes later ConfigMap changes through its environment; an intentional Pod replacement is the documented mechanism for rolling out the new value. Without one, ordinary scaling can leave a service running a mixture of old and new values. Container recreation can introduce an even narrower form of skew because the kubelet resolves referenced values when it creates each container.

Secret synchronization adds another independent clock. The External Secrets Operator, for example, supports CreatedOnce, Periodic, and OnChange refresh policies. Periodic rereads the external provider on an interval. OnChange reacts to changes in the ExternalSecret resource rather than provider-side rotation. CreatedOnce normally creates the target once, but reconciles it again if the target Secret is deleted or altered; recreating the ExternalSecret also begins a new lifecycle. “The controller is installed” therefore tells us very little about when a particular change should materialize.

The wider lesson is not specific to Kubernetes: every edge in the graph has a propagation policy. It might be push-based, polled, rollout-bound, startup-only, manually triggered, or immutable. If that policy is left implicit, operators will invent one in their heads, and their models will disagree during an incident.

A useful configuration specification answers four questions for every edge:

• What event starts propagation?
• What is the expected delay?
• How is failure surfaced and retried?
• What evidence proves the next stage accepted the revision?

Without those answers, eventual consistency becomes “probably updated by now.”

The consumer set is larger than it looks

Teams usually wire configuration into the workload that uses the feature. That sounds precise and least-privileged. It can still be incomplete.

Suppose the web process uses EVENT_SIGNING_KEY, while a worker never signs an event. If both processes boot through one configuration schema that marks the key as globally required, the worker is also a consumer of that configuration for startup purposes. It can fail before reaching any feature-specific code.

This creates an architectural choice.

One option is to deliver the value to every workload that runs the global validator. That restores the runtime invariant, but it widens secret distribution. Another is to split configuration schemas by process role so that each workload validates only what it can use. That preserves least privilege, but introduces more schemas and makes shared startup paths less uniform. A third is conditional validation: keep a shared schema, but require a field only when the relevant capability is enabled, then assert the invariant again at the feature boundary. This limits distribution while keeping common parsing, at the cost of a more stateful contract.

None of these choices is universally correct. The mistake is using one definition of consumer in application code and another in deployment configuration.

For delivery purposes, the consumer set includes every workload that either uses the value or requires it to start. That includes web processes, workers, scheduled jobs, migration hooks, administrative commands, and sometimes sidecars. Short-lived jobs are easy to miss because they may not exist when the change is inspected, yet the next deployment can create them with stale wiring.

Important For every workload that consumes or startup-validates a configuration field, there must be a complete delivery path from its declared source to activation in that workload.

This is a graph property, not a check that a key appears somewhere in a repository.

Version skew is normal

A rolling deployment intentionally runs old and new Pods at the same time. Kubernetes controls the overlap through maxUnavailable and maxSurge; it does not provide an instant fleet-wide transition. Configuration changes therefore need a compatibility story just as binary changes do.

The dangerous case is a coupled change:

• the old binary accepts configuration schema v1;
• the new binary requires schema v2;
• the configuration is stored and rolled out independently.

Publishing v2 first can break old instances. Deploying the binary first can break new instances that still receive v1. A rolling update creates an overlap in which incompatible binary/configuration combinations are possible unless the release process or delivery mechanism prevents them.

The usual solution is an expand-and-contract sequence:

1. Deploy code that accepts both the old and new configuration shapes.
2. Verify that the compatible binary is active everywhere that matters.
3. Publish the new configuration revision gradually.
4. Verify activation across the fleet.
5. Remove support for the old shape in a later release.

Credential rotation is the same problem with an external participant. If a producer begins signing with a new key before consumers can verify it, or a database password changes before connection pools receive it, rollback may not be as simple as restoring a file. Safe rotation often needs an overlap period in which both revisions are accepted, followed by evidence that the old revision is no longer in use.

This is why configuration deserves versioning. A version is not only history in Git. It is an identity that can travel through the graph and let us compare desired, materialized, delivered, and activated state.

Validate at the boundary that owns the failure

“Fail fast” is good advice until every layer fails for the same reason and nobody can tell which boundary was broken.

Validation should be layered.

Before deployment, validate syntax, schema, generated manifests, references, and the expected consumer set. These checks catch structural errors without needing production credentials.

During reconciliation, report whether the desired source was fetched and the target object was produced. A controller condition should distinguish missing permissions, missing source data, transformation errors, and an intentionally non-refreshing policy.

At process startup, validate required presence, parsing, ranges, mutually dependent fields, and invariants the application can determine locally. A process should not advertise readiness if it cannot serve correctly with its activated configuration. Readiness should answer whether this replica can serve safely now, not whether the fleet has converged on the desired revision.

During runtime reload, build and validate a complete candidate before swapping it into use. Updating fields one at a time exposes combinations that never existed in any declared revision. If a candidate is invalid, continuing with the last-known-good revision is often safer than partially activating it.

That last rule has limits. Continuing with an expired or revoked credential may be unsafe, and retaining configuration indefinitely can hide a broken control plane. Last-known-good behavior therefore needs an expiry policy and an explicit degraded state, not silent immortality.

Validation also needs semantic checks. A value can be present, correctly typed, and operationally disastrous. A timeout of 0 may parse as an integer but disable a safety boundary. A percentage of 100 may be valid syntax but turn a gradual rollout into a global change. The component that understands the meaning should own those invariants.

Observe revisions, not values

During an incident, an operator should be able to answer:

• What revision is desired?
• What revision did the synchronization layer materialize?
• What revision did each workload activate?
• When did each transition happen?
• Which consumers are missing, stale, or rejecting the revision?

Secrets and other sensitive configuration values should not appear in logs or metrics. Bounded, explicitly non-sensitive effective settings can be useful for diagnosis, but should be selected deliberately rather than dumped wholesale. Hashing is not automatically safe either: hashes of low-entropy configuration can be guessed, and a unique revision label attached to every metric series creates unbounded cardinality.

A better design carries an opaque revision identifier alongside the value. Prefer an identifier assigned by the authority; if a process must derive one, use a keyed digest rather than a bare hash of the configuration. Controllers can record it in status or metadata. Processes can emit it once in a structured startup or reload event and expose it through an authenticated, authorized, and bounded diagnostic inventory. Fleet-level monitoring can then report counts such as “27 of 30 consumers activated the desired revision” and the age of the oldest stale consumer without turning every revision into a permanent time series.

Health and configuration state should also remain distinct. A service may be healthy on its last-known-good revision while configuration convergence is degraded. Collapsing both into one boolean either removes a healthy instance unnecessarily or hides propagation failure. Report service readiness, configuration activation, and convergence separately, then decide which conditions should block rollout.

Roll out configuration like code

Configuration can change behavior without changing a binary, which sometimes makes it more dangerous than code. It often receives less testing precisely because it looks smaller.

Google’s SRE guidance for safe configuration changes identifies three useful properties: gradual deployment, rollback, and automatic rollback—or at minimum stopping rollout progress—before a failure removes operator control. Those requirements follow naturally from the distributed-system model.

Gradual rollout limits the number of consumers on a bad revision. A pause between stages creates time to inspect both application outcomes and convergence. Rollback requires preserving the previous revision and knowing whether reversing configuration is sufficient; external state changes may require a forward repair instead. Automatic stopping requires signals tied to the effect of the configuration, not only confirmation that distribution succeeded.

The release unit matters too. If a configuration revision is meaningful only with one binary version, deploy them as one tested artifact or encode their compatibility explicitly. If configuration must remain independently changeable, design overlapping compatibility windows. Independence in the delivery machinery does not create independence in semantics.

Test the graph

Most configuration tests focus on the endpoints. A schema test proves the application can parse an example. A deployment test proves a Secret exists. The failure-prone part is the path between them.

Useful tests cover the edges:

• the declared source key maps to the intended materialized key;
• every workload in the consumer inventory references the right object;
• generated manifests preserve the reference in every environment;
• startup validation accepts the deployed shape;
• old and new binary/configuration combinations behave as planned;
• rotation, reload, rollback, and last-known-good paths work;
• short-lived jobs and hooks receive the same contract as long-running services;
• observability reports the activated revision without exposing data.

Not all of these tests need a cluster. Rendering manifests and comparing them with a declared consumer inventory catches many omissions deterministically. Contract tests can use non-secret placeholders to exercise mappings and parsers. A small integration environment can then verify reconciliation, Pod replacement, reload behavior, and convergence.

The goal is not to reproduce production in CI. It is to prove that every edge has an owner and at least one test at the cheapest layer capable of finding its failures.

Configuration is deployed state

Configuration is often described as code, which usefully brings review and version control to it. But code in a repository is not running software, and configuration in its authority is not activated state.

Configuration is a propagation graph, and its lifecycle spine is:

flowchart LR
    Declared["Declared"] --> Materialized["Materialized"]
    Materialized --> Delivered["Delivered"]
    Delivered --> Activated["Activated"]
    Declared -.-> Evidence["Evidence plane"]
    Materialized -.-> Evidence
    Delivered -.-> Evidence
    Activated -.-> Evidence

    classDef lifecycle fill:#efe7ff,stroke:#6a3fd4,color:#20113a
    classDef evidence fill:#d8f3ef,stroke:#1b8c7a,color:#0f3c36
    class Declared,Materialized,Delivered,Activated lifecycle
    class Evidence evidence

The configuration lifecycle. Solid arrows move a revision toward runtime activation; dotted arrows record the evidence needed to prove where it reached. Real systems fan out into many materializations and consumers around this spine.

Every arrow can delay, retry, transform, reject, or silently retain an older revision. Every fan-out can reach some consumers and miss others. Most rolling or staged rollouts create a period of version skew. Every rollback depends on whether the old world still exists outside the configuration system.

Once those properties are explicit, configuration incidents become less mysterious. Instead of asking whether the value exists, ask which revision each stage holds. Instead of checking one workload, enumerate the consumers. Instead of assuming a rollout makes change atomic, design compatibility for the overlap. Instead of logging secrets, propagate revision identity.

A configuration change is finished when all intended consumers have activated the intended revision and the system can prove it. Everything before that is propagation in progress.

References

• Updating Configuration via a ConfigMap — Kubernetes
• Deployments — Kubernetes
• ExternalSecret API — External Secrets Operator
• Configuration Design and Best Practices — Google SRE Workbook

Two interactive animations do most of the explaining. Each has a guided tour you can step through beat by beat, plus a free mode where you fire operations and watch them move.

Keyboard in both: ↓/↑ step one beat, ←/→ jump sections, Space play or pause, R replay, Home/End first or last section.

The V8 heap

The model in one paragraph. Most objects live in a heap split into spaces. A small New space, itself split into two equal halves, takes most fresh allocations via a bump pointer; anything above the large-object threshold, or covered by pretenuring, skips it. A large Old space holds whatever survived long enough to be promoted. Around them sit Large-object space for things too big to copy, Code space for JIT output, and an immortal Read-only space.

The parts that are easiest to get wrong:

Not everything is on the heap. Small integers are encoded directly in the value and never allocated at all. ArrayBuffer backing stores, including Buffer bytes, live outside V8’s managed heap, which is why --max-old-space-size does not bound them. process.memoryUsage().arrayBuffers reports them specifically, counted within the broader external figure. This is the single most common surprise when diagnosing real memory growth.

Minor GC is cheap because of what it does not do. The Scavenger copies live objects from one semi-space to the other, then flips their roles. Dead objects are never individually traced or copied; their half is reclaimed wholesale. Cost tracks the survivors, not the allocation volume. It is still a stop-the-world pause, just a short parallel one.

Promotion is positional, not a counter. The rule of thumb is that surviving a second scavenge promotes you to Old space. The mechanism is not a per-object counter, it is an age mark in the semi-space: a reachable survivor sitting below that mark is eligible for promotion. Copy or space pressure can promote survivors earlier, and allocation-site pretenuring is a different thing entirely, since it allocates straight into Old space and skips New space altogether. Treat “survives twice” as the common path, not a guarantee.

Marking is transitive, which is the whole point. Major GC is Mark-Compact. Objects start white, become grey when discovered, and turn black once their fields have been scanned. Scanning a grey object greys everything it references, so an object stays alive by being reachable at any depth, not by being named directly by a root. The invariant that no black object points to a white one is what guarantees nothing live is missed. The animation walks this one edge at a time, which is the part flat diagrams never show.

Sweep and compact are different phases. Sweeping adds the gaps left by dead objects in the paged spaces to the free-lists. Unreachable large objects are reclaimed in that same phase, but by releasing their pages outright rather than through a free-list. Compaction is selective: V8 evacuates live objects off its most fragmented pages onto fresh ones. Large objects are never relocated, so they are freed in the sweep but are never part of compaction.

Orinoco shrinks pauses, it does not remove them. Incremental marking splits work into small scheduled steps, concurrent marking runs on helper threads while JavaScript executes, and parallel collection uses several threads to finish the unavoidable pauses faster. V8’s write-barrier machinery is what makes that safe, and it serves at least three purposes: recording old-to-young references in remembered sets for the minor GC, preserving the marking invariant so concurrent marking cannot miss an object that becomes reachable mid-flight, and recording old-to-old slots that will need their pointers updated after compaction moves things.

One heap per isolate. That fact is what makes the second half make sense.

The Node event loop

Node runs your JavaScript on one thread. Everything else is about how work reaches that thread.

The phase order moved. This is the one worth double-checking against whatever you already believe. Since libuv 1.45, which landed in Node 20.3.0 (Node 20.0 through 20.2 still carried libuv 1.44.2 and ran timers first), a loop iteration runs:

pending → idle/prepare → poll → check → close → timers

Timers run at the end of an iteration, not the start, and a compatibility timer pass runs each time the loop is entered in default mode. Note also that pending gets a second turn right after poll, up to eight times, to avoid starving that queue.

Most third-party diagrams still show timers first. Node’s own guide has since been corrected and now documents the change explicitly, with timers appearing twice in its cycle diagram.

Drawn as an endless cycle the phase sequence looks unchanged, but do not read that as “nothing happens differently”. The move is observable. A setTimeout(0) scheduled from inside an I/O callback used to wait for the next iteration’s timers phase; now the timers phase is still ahead of it in the same iteration, so it can fire without another trip through poll. Node’s guide says as much: the change can affect how timers and setImmediate interact.

Where the waiting happens. The loop can wait in the poll phase when nothing is immediately runnable but the loop is still alive, meaning it holds referenced handles or requests. Note that liveness is not only sockets: a pending timer is itself an active handle and will keep the loop waiting. The wait is bounded by the nearest timer deadline, or unbounded if no timer is armed. It does not wait at all if a setImmediate is queued, which Node arranges with an idle handle whose only job is to stop the loop blocking. If nothing referenced remains, the loop exits and the process ends.

The usual folk version of this is “it blocks when there is nothing to do”. The empty-queue half is right; the “nothing to do” half is exactly wrong, because with no referenced work left the loop does not block, it exits.

Two queues sit outside the phases. When a callback hands control back to Node, the process.nextTick queue drains completely, then V8’s microtask queue drains completely, before the loop advances. Since Node 11 this happens between individual timer and immediate callbacks too, not only between phases.

Both names are traps. process.nextTick does not wait for the next iteration; it runs before the loop advances at all. setImmediate targets the check phase right after poll, which may still be ahead of you in the current iteration rather than the next one, depending on where you scheduled it. And the nextTick-before-promises rule holds at a CommonJS callback boundary but inverts in top-level ESM, where the module body is itself evaluated inside a microtask drain, so promise callbacks scheduled there run before nextTick.

What actually uses the thread pool. Most async fs, dns.lookup(), async zlib, and specific crypto calls such as pbkdf2, scrypt, randomBytes and key generation. Native addons can queue work there too. Socket readiness does not use it, and neither does dns.resolve*(), which does its own network queries. This is why Node can hold tens of thousands of connections even though the pool has only four threads. Do not conclude that network code never touches the pool though: anything connecting by hostname usually goes through dns.lookup() first, so name resolution can contend for those same four threads.

The pool defaults to 4, is capped at 1024, and is set by UV_THREADPOOL_SIZE before the process starts. It is process-global: every event loop in the process, worker threads included, shares the same threads. Spawning workers does not multiply it. Fire more pool-backed work than there are threads and it queues, and one long task effectively shrinks the pool by one.

Three unrelated sets of threads get conflated constantly: V8’s own GC and JIT helpers, libuv’s process-global pool of four, and any worker threads you spawn. Only the last runs your JavaScript.

Workers are isolates, not processes. Each worker gets its own V8 isolate, heap, event loop, and pair of queues, so ordinary objects can never be shared. For payload memory there are three choices: postMessage structured-clones a copy, a transfer list moves an ArrayBuffer’s ownership with no copy, and a SharedArrayBuffer maps the same bytes into both isolates with Atomics to coordinate. Transfer lists can also hand over resources such as a MessagePort or a FileHandle. Separate heaps also means separate GC, so a worker’s collection does not pause the main thread.

Workers are not a full isolation boundary though. An uncaught exception terminates that worker and emits error on its parent-side Worker object, and the parent survives only if it handles that event: an unhandled error on an EventEmitter is thrown and will take the process with it. A native fault or a global OOM takes everything down regardless. resourceLimits caps selected per-worker JS-engine resources rather than the worker’s total memory, and notably excludes external data such as ArrayBuffer backing stores, which loops back to the external-memory point above.

Processes are the real boundary. With cluster, a primary process forks however many workers you ask for, commonly one per os.availableParallelism(). Under the default off-Windows SCHED_RR policy the primary owns the listening socket and hands accepted connections to workers over IPC; Windows lets the OS distribute them instead. Each process gets its own isolate, heap, event loop and --max-old-space-size budget, so a GC pause in one never stalls another. They still share OS resources such as files, sockets and system limits, so the isolation is strong but not absolute.

Where the two halves meet

The reason to care about Orinoco is that in Node a stop-the-world GC pause lands on the event loop. Callbacks queued on that isolate’s loop wait behind it, exactly as they would behind a synchronous JSON.parse of something enormous. Other worker isolates and other processes keep running, and concurrent marking is not itself a pause. Still, the list of things that block your loop is not just your own slow code; it includes the collector.

Which also explains why the process-per-core shape is more than a throughput trick. N processes means N independent heaps, each collected on its own schedule, so one process pausing to collect does not stall the others. Worker threads give you the same separation of heaps inside a single process, at the cost of sharing that process’s fate.

One closing warning. The phase order is the detail most likely to be stale in your head, because the diagram everyone memorised outlived the implementation by several major versions. Widely repeated diagrams drift, and so do assumptions about which sources have caught up. Reading uv_run() settles it in about a minute.

Every public function, type, default value, error message, dependency, and example becomes something users can build on. Once they do, changing it is no longer a private refactor. It is a negotiation with people who are trying to keep their own systems working.

That is the part that took me time to appreciate. A library can be small, elegant, and well-tested, but the moment it has users, its shape matters beyond its implementation. The API becomes a promise. The docs become a promise. The release process becomes a promise. Even the things you choose not to expose become part of how users understand the library.

I have felt this across different kinds of projects: a larger API client such as openai-kotlin, smaller focused Kotlin libraries, and newer runtime-boundary work like execbox. The details are different, but the maintenance pressure is the same: once people depend on your library, you are not only publishing code. You are publishing expectations.

Public API is more than types

It is tempting to think of public API as the list of exported symbols. Functions, classes, interfaces, modules, packages. That is only the visible part.

The real API also includes behavior:

• what happens when input is missing,
• whether calls are lazy or eager,
• which errors are thrown and when,
• whether ordering is stable,
• how cancellation works,
• how retries, timeouts, and defaults behave,
• which platforms are supported,
• which values are accepted even if the type allows more.

Users learn those details from your implementation, docs, examples, and tests. If the behavior is useful, they will depend on it. If the behavior is accidental, they may still depend on it.

This is why library code needs a different level of care than application code. In an application, an internal function can be fixed when its caller changes. In a library, you do not control the callers. They live in other repositories, other companies, other release cycles, and sometimes other time zones. You only see them when an issue appears.

The harder part is that users rarely depend on your API exactly the way you imagined. They will compose it with frameworks you do not use, run it on platforms you do not test every day, and rely on edge cases because those edge cases solve real problems for them.

That does not mean every behavior must be frozen forever. It means public behavior should be intentional. If something is not meant to be stable, do not expose it casually. If something is stable, test it like a contract.

Small surfaces survive

The easiest API to maintain is the one you did not publish.

This sounds obvious, but it is one of the most useful lessons in library design. Every public helper, option, overload, type alias, package path, and configuration hook creates future work. It may need documentation. It may need tests. It may need compatibility. It may limit a future internal design.

Small public surfaces are not about minimalism for its own sake. They are about preserving room to improve the library without breaking users.

A good library usually has more internal machinery than public API. That is fine. Internals can be ugly for a while. They can be renamed, split, optimized, generated, deleted, or replaced. Public API has a different cost model. Once it exists, removal is expensive.

This is especially important when a project is young. Early versions are full of uncertainty. You may not know the right abstractions yet. You may not know whether users need a low-level primitive or a higher-level workflow. Publishing too much too early turns guesses into obligations.

I saw this while preparing execbox for a stable library surface. Experimental runtime packages had accumulated around the core idea, so I removed them before 1.0 and kept two supported responsibilities: provider contracts in @execbox/core and QuickJS execution in @execbox/quickjs. That cleanup was disruptive in the short term, but it made the public promise smaller and clearer.

The better default is to expose the smallest useful path, then let real usage pull more surface area out of the internals. When a pattern repeats, promote it. When users keep reaching around the API, understand why. When an option exists only because the implementation happened to have it, keep it private.

Small surfaces also make documentation and examples better. A library that can be explained with a few concepts is easier to adopt, easier to debug, and easier to trust.

Docs are part of the contract

Documentation is not a decoration around the library. For many users, it is the library.

The first example teaches them what the maintainers consider normal. The getting-started page defines the happy path. The advanced guide tells them which use cases are expected. The upgrade guide tells them whether changes are predictable. Missing docs tell them where the contract is weak.

This is why examples matter so much. Users copy them. They build habits from them. If the example skips error handling, people will skip error handling. If it uses an unstable internal helper, people will use that helper. If it shows a pattern that only works in a narrow environment, users will assume the library failed when it does not work elsewhere.

Docs also help maintainers make decisions. If a behavior cannot be explained clearly, the API may be wrong. If a feature needs five paragraphs of caveats, it may be too complex, too early, or sitting at the wrong abstraction level. If the docs keep saying what the library does not do, the project may not have a clear enough positive shape yet.

The best docs do not need to cover every implementation detail. They need to make the contract legible:

• what the library is for,
• what the main path looks like,
• which guarantees users can rely on,
• where the boundaries are,
• how to upgrade when those boundaries move.

In practice, docs and design feed each other. Writing the docs often exposes where the API is too clever, where a type name is vague, or where a feature has no obvious place in the mental model.

Compatibility is a budget

Compatibility is not binary. It is a budget you spend.

Every breaking change spends user trust. Sometimes that spend is worth it. Bad APIs should not live forever just because they were published once. A confusing abstraction can cost users more over time than a well-explained migration. Pre-1.0 libraries especially need room to correct their shape before stability hardens the wrong design.

But breaking changes should be honest. They should solve a real problem, not clean up maintainer discomfort. They should come with migration notes. They should avoid surprising users with unrelated churn. They should be grouped carefully instead of scattered across releases without a story.

Deprecation is useful when it gives users time to move. It is less useful when it becomes a permanent museum of old ideas. A deprecation should answer three questions:

• what should users do instead,
• when does the old path go away,
• why is the change worth making.

For projects that follow Semantic Versioning, version numbers help here, but they are not enough by themselves. SemVer treats the public API as unstable during 0.y.z and uses major-version changes after 1.0 to signal incompatible API changes. Neither signal tells users whether the change is understandable, whether the migration is realistic, or whether the maintainers respect their time.

Compatibility also includes softer promises: supported platforms, runtime versions, generated code shape, dependency ranges, serialization formats, package names, module paths, and error semantics. These are easy to treat as implementation details until users build on them.

The maintainer’s job is not to avoid all change. It is to make change predictable.

Dependencies become user dependencies

Every runtime dependency you expose, including its transitive dependencies, becomes part of someone else’s application.

That does not mean libraries should have no dependencies. Good dependencies can reduce bugs, improve standards compliance, and let maintainers focus on the library’s actual purpose. But dependencies carry costs that are different in a library than in an application.

An application chooses its own runtime, deployment target, bundle size, dependency policy, and upgrade schedule. A library is pulled into environments it does not control. A transitive dependency can affect build time, binary size, cold start, platform support, security reviews, licensing, and version resolution.

The question is not “can this dependency help?” The question is “is this dependency part of the contract I want users to inherit?”

Sometimes the answer is yes. Sometimes the answer is no. Sometimes the right design is to keep an integration optional, put it behind a separate package, or accept a little more local code to avoid forcing a large dependency onto every user.

Dependencies also shape maintenance. If your library wraps a fast-moving API, generated models or protocol clients may be necessary. If your library targets multiple platforms, dependency choices can decide which platforms remain possible. If your library sits close to runtime boundaries, dependency behavior can leak into security, lifecycle, or performance expectations.

The dependency tree is not invisible. Users will feel it.

Maintenance is product work

Maintaining a library is product work under technical constraints.

The product is not a UI. It is the experience of adopting, understanding, upgrading, debugging, and trusting the library. Issues, pull requests, release notes, examples, CI, package metadata, and error messages are all part of that experience.

This is where taste matters, but not in the vague sense. Taste is choosing boring names when clever names would be memorable. It is saying no to an option that would make one user happy but weaken the model for everyone. It is keeping a release small enough that users can understand it. It is accepting that a missing feature is sometimes better than a feature with the wrong contract.

A maintainer has to balance different kinds of pressure:

• new users want the simplest possible start,
• advanced users want escape hatches,
• contributors want their use cases accepted,
• existing users want stability,
• the maintainer wants the codebase to remain workable.

Those goals conflict. A healthy library does not satisfy all of them equally. It chooses a center of gravity and makes that choice visible.

For me, this is the main difference between writing code and writing libraries. Code can be correct in isolation. A library has to be correct in relation to users. It has to age. It has to carry old decisions until they can be changed responsibly. It has to leave enough space for future maintenance.

The quiet goal

The quiet goal of a library is predictability.

Users should be able to predict how the API behaves. They should be able to predict whether an upgrade is risky. They should be able to predict where to look when something fails. They should be able to predict whether a feature belongs in the library or outside it.

That predictability does not happen by accident. It comes from treating the public surface as a contract, keeping that contract small, documenting it clearly, changing it deliberately, and remembering that every dependency and release is part of the user’s system too.

Reusable code is the beginning. The real work is making it safe for other people to build on.

That changes the shape of the system. Tool definitions do not all have to sit in the model context. Intermediate results do not all have to be replayed through the model. Multi-step logic can run closer to the data it is manipulating.

This is the pattern execbox is built around: a reusable Node.js library layer for exposing host-defined tools and wrapped MCP servers to guest JavaScript, while keeping capability and runtime boundaries explicit.

Problem

Direct MCP tool loops are a good default. The client exposes tools, the model picks one, the host executes it, the result goes back into context, and the model decides what to do next.

That loop is simple, but an eager implementation scales poorly once the tool catalog or intermediate data gets large:

• every tool definition exposed up front consumes context,
• every intermediate result passes back through the model,
• large payloads are copied and summarized repeatedly,
• multi-step control flow becomes token-heavy.

flowchart LR
    M["Model"] --> T["Tool catalog in context"]
    T --> C1["Call tool A"]
    C1 --> R1["Return result to model"]
    R1 --> C2["Call tool B"]
    C2 --> R2["Return another result"]
    R2 --> M

    classDef model fill:#efe7ff,stroke:#6a3fd4,color:#20113a
    classDef catalog fill:#fff3d6,stroke:#d1a11f,color:#4f3200
    classDef tool fill:#d8f3ef,stroke:#1b8c7a,color:#0f3c36
    class M model
    class T catalog
    class C1,R1,C2,R2 tool

For tools that return large documents, search results, database rows, logs, or API payloads, the loop spends too much of the model budget on mechanical data movement. A compact programming surface lets the model call tool-like APIs, filter intermediate values locally, and return only the final result the host needs to see.

Signals

Posts from Anthropic and Cloudflare, along with the MCP client best practices, describe the same architecture pressure.

Anthropic’s post, Code execution with MCP: Building more efficient agents, frames direct MCP usage around two scaling problems: tool definitions consume context, and intermediate results consume more context. Their answer is to let the model write code against tool-like APIs, load definitions on demand, and keep intermediate processing inside the execution environment.

Cloudflare’s post, Code Mode: give agents an entire API in 1,000 tokens, makes the same argument from the API side: a large tool surface can become a smaller typed SDK surface that the model uses from generated code. Cloudflare then followed with Sandboxing AI agents, 100x faster, focused on where generated code should run.

The MCP docs call this pattern Programmatic Tool Calling / Code Mode: the model writes code, the code runs in a sandbox, and the host brokers MCP tool calls so only the final result needs to return to the model.

The two scaling pressures have related but distinct answers. Progressive discovery controls which tool definitions enter the model context, while programmatic tool calling controls how tools are invoked and where intermediate results are processed. They can be used independently or together.

Together, these posts and docs point in the same direction: eager direct tool calling is useful but expensive at scale, code execution can compress data movement, and the runtime cannot be an afterthought.

Execbox

execbox is the library layer I wanted for that pattern. It is not an agent framework or hosted sandbox product; it is a set of Node.js packages that turn host capabilities into callable guest namespaces, then run guest JavaScript against those namespaces through a chosen executor.

The package map is intentionally small: @execbox/core owns the execution contract, provider resolution, and MCP adapters; @execbox/quickjs provides inline and worker-hosted QuickJS execution.

The core flow stays the same across those packages: host code defines tools or discovers them from MCP, those tools become a deterministic guest namespace, guest code runs against that namespace, tool calls cross a host-controlled boundary, and results come back as JSON-compatible data. The same guest code shape can start with inline QuickJS and move to worker-hosted QuickJS without changing the provider contract.

sequenceDiagram
    autonumber
    participant App as Host application
    participant NS as Resolved namespace
    participant Guest as Guest runtime
    participant Boundary as Host boundary
    participant Systems as Systems / APIs / MCP servers

    App->>NS: Define or discover capabilities
    App->>Guest: Execute code with namespace
    Guest->>Boundary: Call tool
    Boundary->>Systems: Invoke capability
    Systems-->>Boundary: Structured result
    Boundary-->>Guest: Return JSON-safe value
    Guest-->>App: Return execution result

MCP can appear on either side of the flow. Upstream MCP servers can be wrapped into guest namespaces, and execbox can also expose code execution itself as an MCP server so a client gets a compact code-running surface instead of a large direct tool catalog.

Usage

In TypeScript, a typical MCP provider flow starts with an MCP server declared through the MCP SDK, then wraps it as an execbox provider.

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { openMcpToolProvider } from "@execbox/core/mcp";
import { QuickJsExecutor } from "@execbox/quickjs";
import * as z from "zod";

const upstreamServer = new McpServer({
  name: "upstream",
  version: "1.0.0",
});

upstreamServer.registerTool(
  "search-docs",
  {
    description: "Search documentation.",
    inputSchema: { query: z.string() },
    outputSchema: { hits: z.array(z.string()) },
  },
  async (args) => ({
    content: [{ text: `found ${args.query}`, type: "text" }],
    structuredContent: { hits: [args.query] },
  }),
);

const handle = await openMcpToolProvider({ server: upstreamServer });

try {
  const executor = new QuickJsExecutor();
  const result = await executor.execute(
    '(await mcp.search_docs({ query: "quickjs" })).structuredContent.hits[0]',
    [handle.provider],
  );

  if (!result.ok) {
    throw new Error(result.error.message);
  }

  console.log(result.result);
} finally {
  await handle.close();
}

The runtime choice is separate from the provider shape. Use inline QuickJS for trusted, lowest-friction local execution. Use worker-hosted QuickJS when you want local execution off the main thread with worker lifecycle controls. Both use the same provider and execution contracts; the worker changes runtime placement and lifecycle, not the capability set.

Boundaries

The runtime does not own the capabilities. The provider and exposed tool surface define them.

If guest code can call a tool that deletes data, sends email, or reaches a private system, then guest code has that authority. Moving execution from inline QuickJS to a worker changes lifecycle and runtime placement, not what the exposed tools are allowed to do.

That capability boundary is not a runtime authorization decision. The host still needs to evaluate each sandbox-originated tool call against the applicable user confirmation or categorical grant. Approving a generated script should not automatically authorize every call it makes.

Execbox helps make that execution path controlled: fresh execution state per call, JSON-only tool and result boundaries, schema validation around host tool execution, bounded logs, timeout and memory controls, and abort propagation into in-flight host work.

Those controls matter, but they do not make a dangerous tool safe to expose. They make it easier to expose only the tools you intend, run generated code through a stable contract, and choose the runtime placement that matches the deployment.

That is the role of execbox: keep one capability model, support MCP tools and wrapped MCP servers, and let applications choose between inline and worker-hosted QuickJS without rewriting the guest/tool contract.

If you want to look at the implementation:

• Getting Started
• Examples
• Architecture
• Security
• GitHub repository

The problem

In Java, a program code can change a lot between its Java source code form, Byte code form, and machine code form. The Java source code focuses more on readability and clarity, while the machine code focuses on performance and efficiency. The JVM is allowed to optimize the code, with different degrees of optimization (depending on the compilation stage), as long as it remains correct. But, this task can be handy in the context of multi-threaded applications.

Sequential consistency

There are multiple levels of caching while executing a program; the processor never operates on values directly in the main memory, but instead, it loads the values to its cache, manipulates them, then writes them back to the main memory. Let’s take the following example:

class Reorder {
    int foo = 0;
    int bar = 0;

    void calc() {
        foo += 1; //#1
        bar += 1; //#2
        foo += 2; //#3
    }
}

How the processor can execute the method calc() in memory?

1. Load foo from main memory to processor cache. Increment by 1, write it back the main memory (#1).
2. Load bar from main memory to processor cache. Increment by 1, write it back the main memory (#2).
3. Load foo from main memory to processor cache. Increment by 2, write it back the main memory (#3).

How the earlier example can be optimized ? by swapping the instructions (#2 and #3):

void calc() {
    foo+=1; //#1
    foo+=2; //#3
    bar+=1; //#2
}

1. Load foo from main memory to processor cache. Increment by 1, Increment by 2, write it back the main memory (#1 and #3).
2. Load bar from main memory to processor cache. Increment by 1, write it back the main memory (#2).

In a single-threaded program, this optimization can be considered without side effects, however, in a multi-threaded world, it introduces some abnormal behavior:

The possible values of the variables overtime in the two cases shows the slight difference:

• Before optimisation:
  1. (foo == 0, bar == 0)
  2. (foo == 1, bar == 0)
  3. (foo == 1, bar == 1)
  4. (foo == 3, bar == 1)
• After optimisation:
  1. (foo == 0, bar == 0)
  2. (foo == 1, bar == 0)
  3. (foo == 3, bar == 0)
  4. (foo == 3, bar == 1)

This previous example is an optimization that the JVM is allowed to do. The JVM can do much more complex optimizations, however, the outcome might be unexpected in a multi-threaded world! But why optimize then? The answer: memory access latency!

Eventual consistency

A machine can have multiprocessors, and (at some level) each processor has its cache, which means, each processor loads only the values it needs for its operations. Let’s say we have two processors and the following program:

class Caching {
    boolean flag = true;
    int count = 0;

    void thread1() {
        while (flag) count++;
    }

    void thread2() {
        flag = false;
    }
} 

Let’s say processor #1 will run the method thread1() and processor #2 will run the method thread2(). An optimization can be the following:

• Since thread1() never modifies the flag variable, there is no need to load it from the main memory for each loop check, only once to the cache is enough -> the changes to flag might never be observed!
• Processor #2 has no obligation to write it changes to the flag variable to the main memory! This means an optimization can be to simply not do the operation at all!

1.3 Atomicity

The atomicity in Java is to consider all values are atomic, which means that the modification to a variable (for example 64 bit types like long and double) to be done atomically.

Let’s consider the following example:

class LongTearing {
    long foo = 0L;

    void thread1() {
        foo = 0x0000FFFF; // 2147483647 
    }

    void thread2() {
        foo = 0xFFFF0000; // -2147483648 
    }
} 

A 64 bit long variable, is written in two slots in the case of 32 memory, a problem can occur here:

• thread1() writes the first half of its value to memory 0000.
• thread2() writes the second half of its value to memory 0000.
• thread1() writes the second half of its value to memory FFFF.
• thread1() writes the first half of its value to memory FFFF.
• The final value of the variable will be then: 0xFFFFFFFF !!!

Processor optimization

Ordering operations sometimes are tied to the processor architecture. Optimization needs can be different for example between ARM processors and x86 processors. ARM processors can be more aggressive because they are designed for energy-consuming efficiency, than x86 processors which are more about calculation speed.

What is the Java memory model?

Java memory model answers the question: what values can be observed upon reading from a specific field?

Formally specified by breaking down a Java program into actions and applying several orderings to these actions. If one can derive a so-called happens-before ordering between a write action and a read action of one field, the Java memory model guarantees that the read returns a particular value.

The Java memory machine guarantees intra-thread consistency equivalent to sequential consistency.

Building blocks

According to the Java memory model, using the following keywords, a programmer can indicate to the JVM to refrain from optimizations that could otherwise cause concurrency issues:

• Field-scoped: final, volatile.
• Method-scoped: synchronized (method/block), java.util.concurrent .*.

In terms of the Java memory model, the above concepts introduce additional synchronization actions which introduce additional (partial) orders. Without such modifiers, reads and writes might not be ordered what results in a data race.
A memory model is a trade-off between a language’s simplicity (consistency/atomicity) and its performance.

Volatile

Let’s take the following example:

class DataRace {
    boolean ready = false;
    int answer = 0;

    void thread1() {
        while (!ready) ;
        assert answer == 42;
    }

    void thread2() {
        answer = 42;   // #1
        ready = true;  // #2
    }
} 

The lines #1 and #2 can be reordered! This means, the assertion in method thread1() can fail in a multi-threaded world!
A solution ? The keyword volatile:

class DataRace {
    volatile boolean ready = false;
    int answer = 0;

    void thread1() {
        while (!ready) ;
        assert answer == 42;
    }

    void thread2() {
        answer = 42;   // #1
        ready = true;  // #2
    }
} 

volatile implies for two threads with a write-read relationship on the *same field*, certain optimizations are not allowed!

1. When a thread writes to a volatile variable, all of its previous writes are guaranteed to be visible to another thread when that thread is reading the same value.
2. Both threads must align “their” volatile value with that in main memory (flush).
3. If the volatile value was a long or a double value, word-tearing was forbidden.

Synchronized

Another way to achieve the synchronization is by using: synchronized Let’s check the following example assuming the second thread acquires the lock first:

class DataRace {
    boolean ready = false;
    int answer = 0;

    synchronized void thread1() {
        while (!ready) ;
        assert answer == 42;
    }

    synchronized void thread2() { //Assuming this is called 1st
        answer = 42;
        ready = true;
    }
} 

When a thread releases a monitor, all of its previous writes are guaranteed to be visible to another thread after that thread is locking the same monitor.. This only applies for two threads with an *unlock-lock relationship* on the same monitor!

Thread life-cycle semantics

When a thread starts another thread, the started thread is guaranteed to see all values that were set by the starting thread.

class ThreadLifeCycle {
    int foo = 0;

    void method() {
        foo = 42;
        new Thread() {
            @Override
            public void run() {
                assert foo == 42;
            }
        }.start();
    }
} 

Similarly, a thread that joins another thread is guaranteed to see all values that were set by the joined thread.

Final field semantics

When a thread creates an instance, the instance’s final fields are frozen. The Java memory model requires a field’s initial value to be visible in the initialized form to other threads.

This requirement also holds for properties that are dereferenced via a final field, even if the field value’s properties are not final themselves (memory-chain order).

External actions

A JIT-compiler cannot determine the side-effects of a native operation. Therefore, external actions are guaranteed to not be reordered.

class Externalization {
    int foo = 0;

    void method() {
        foo = 42;
        jni(); // Not re-ordered
    }

    native void jni();
} 

External actions include JNI, socket communication, file system operations, or interaction with the console ( non-exclusive list).

Thread-divergence actions

Thread-divergence actions are guaranteed to not be reordered. This prevents surprising outcomes of actions that might never be reached.

class ThreadDivergence {
    int foo = 42;

    void thread1() {
        while (true) ;
        foo = 0; // Not re-ordered
    }

    void thread2() {
        assert foo == 42;
    }
} 

In the previous example, in the method thread1() the line foo = 0 is unreachable. Thus not re-ordered.

In Practice

The following are some practical examples of Java Memory Model use (or misuse).

Double-checking

The following is a lazy instance creation example:

class DoubleChecked {
    static volatile DoubleChecked instance;

    static DoubleChecked getInstance() {
        if (instance == null) {
            synchronized (DoubleChecked.class) {
                if (instance == null) {
                    instance = new DoubleChecked();
                }
            }
        }
        return instance;
    }

    int foo = 0;

    DoubleChecked() {
        foo = 42;
    }

    void method() {
        assert foo == 42;
    }
} 

This example works because of volatile, omitting it may cause having an instance of an object created, but uninitialized!

Arrays

Declaring an array to be volatile does not make its elements volatile! In the following example, there is no write-read edge because the array is only read by any thread:

class DataRace {
    volatile boolean[] ready = new boolean[]{false};
    int answer = 0;

    void thread1() {
        while (!ready[0]) ;
        assert answer == 42;
    }

    void thread2() {
        answer = 42;
        ready[0] = true;
    }
} 

For such volatile element access: java.util.concurrent.atomic.AtomicIntegerArray.

Sources

• Java memory model - Wikipedia
• JSR-133 Java Memory Model and Thread Specification 1.0 Proposed Final Draft
• Happened-before - Wikipedia
• Java Memory Model - jenkov
• The Java Memory Model for Practitioners
• Close Encounters of The Java Memory Model Kind

Java source codes are compiled into an intermediate state called bytecode (i.e. .class file) using the Java compiler (javac). The Java Virtual Machine a.k.a JVM interprets the bytecode (without further recompilations) into native machine language. Therefore, bytecode acts as a platform-independent intermediary state which is portable among any JVM regardless of underlying OS and hardware architecture.

The JVM is a specification. Vendors are free to customize, innovate, and improve its performance during the implementation.

1. Class Loader Subsystem

The JVM resides on the RAM. During execution, using the Class Loader subsystem, the class files are brought on to the RAM. This is called Java’s dynamic class loading functionality. It loads, links, and initializes the class file (.class) when it refers to a class for the first time at runtime (not compile time).

1.1. Loading

• Bootstrap Class Loader loads standard JDK classes such as core Java API classes (e.g. java.lang.* package classes) from $JAVA_HOME/jre/rt.jar. The class loader acts as parent of all class loaders in Java;
• Extension Class Loader delegates class loading request to its parent, Bootstrap and if unsuccessful, loads classes from the extensions directories (e.g. security extension functions) in extension path $JAVA_HOME/jre/lib/ext or any other directory specified by the java.ext.dirs system property;
• System/Application Class Loader loads application specific classes from system class path, that can be set while invoking a program using -cp or -classpath command line options.

Note: It is possible to directly create a User-defined Class Loader on the code itself.

1.2. Linking

Linking is to verify and prepare a loaded class or interface, its direct superclasses and superinterfaces, and its element type as necessary, while following the below properties:

• Verification: ensure the correctness of .class file, If verification fails, it throws runtime errors (java.lang.VerifyError). For instance, the following checks are performed:
  • consistent and correctly formatted symbol table;
  • final methods / classes not overridden;
  • methods respect access control keywords;
  • methods have correct number and type of parameters;
  • bytecode doesn’t manipulate stack incorrectly;
  • variables are initialized before being read;
  • variables are a value of the correct type.
• Preparation: allocate memory for static storage and any data structures used by the JVM such as method tables. Static fields are created and initialized to their default values, however, no initializers or code is executed at this stage;
• Resolution: replace symbolic references from the type with direct references. It is done by searching into method area to locate the referenced entity.

1.3. Initialization

The initialization logic of each loaded class or interface will be executed (e.g. calling the constructor of a class). Since JVM is multi-threaded, initialization of a class or interface should happen very carefully (i.e. make it thread safe).

2. Runtime Data Areas

Runtime Data Areas are the memory areas assigned when the JVM program runs on the OS.
In addition to reading .class files, the Class Loader subsystem generates corresponding binary data and save the following information in the Method area for each class separately:

• fully qualified class name (FQCN) of the loaded class and its immediate parent class;
• whether .class file is related to a Class, Interface or Enum;
• modifiers, static variables, and method information etc.

For every loaded .class file, it creates exactly one Class object to represent the file in the Heap memory. This Class object can be used to read class level information (class name, parent name, methods, variable information, static variables etc.) later in the code.

2.1 Method Area (Shared)

This is a shared resource (only 1 method area per JVM). All JVM threads share this same method area, which means the access to the method data and the process of dynamic linking must be thread safe.
Method area stores class level data (including static variables) such as:

• ClassLoader reference;
• runtime constant pool;
• field data;
• method data;
• method code.

2.2 Heap Area (Shared)

This is also a shared resource (only 1 heap area per JVM). Information of all objects and their corresponding instance variables and arrays are stored in the Heap area. Heap area is a great target for GC.

2.3. Stack Area (Per thread)

This is not a shared resource (thread safe). Every JVM thread has a separate runtime stack to store method calls. For every such method call, one entry will be created and added (pushed) into the top of runtime stack and such entry it is called a Stack Frame.

A Stack Frame is divided into three sub-entities:

• Local Variable Array: contains local variables and their values;
• Operand Stack: this acts as a runtime workspace to perform any intermediate operation. Each method exchanges data between the Operand stack and the local variable array, and pushes or pops other method invoke results;
• Frame Data: all symbols related to the method are stored here. For exceptions, the catch block information will also be maintained in the frame data.

The frame is removed (popped) when the method returns normally or if an uncaught exception is thrown during the method invocation. Since these are runtime stack frames, after a thread terminates, its stack frame will also be destroyed by JVM.

The stack frame is size fixed, however, the stack itself can be a dynamic or fixed size. If a thread requires a larger stack than allowed a StackOverflowError is thrown. If a thread requires a new frame and there isn’t enough memory to allocate it then an OutOfMemoryError is thrown.

2.4. PC Registers (Per thread)

For each JVM thread, when the thread starts, a separate PC (Program Counter) Register gets created in order to hold the address of currently-executing instruction (memory address in the method area). If the current method is native then the PC is undefined. Once the execution finishes, the PC register gets updated with the address of next instruction.

2.5. Native Method Stack (Per thread)

There is a direct mapping between a Java thread and a native operating system thread. After preparing all the state for a Java thread, a separate native stack also gets created in order to store native method information invoked through JNI (Java Native Interface).

Once the native thread has been created and initialized, it invokes the run() method in the Java thread. When the thread terminates, all resources for both the native and Java threads are released. The native thread is reclaimed once the Java thread terminates. The operating system is therefore responsible for scheduling all threads and dispatching them to any available CPU.

3. Execution Engine

Execution Engine executes the instructions in the bytecode line-by-line by reading the data assigned to Runtime Data Areas.

3.1. Interpreter

The interpreter interprets the bytecode and executes the instructions one-by-one. Hence, it can interpret one bytecode line quickly, but executing the interpreted result is a slower task. The disadvantage is that when one method is called multiple times, each time a new interpretation and a slower execution are required.

3.2. Just-In-Time (JIT) Compiler

The JIT compiler, compiles the bytecode to native code. Then for repeated method calls, it directly provides the native code.

However, even for JIT compiler, it takes more time for compiling than for the interpreter to interpret. For a code segment that executes just once, it is better to interpret it instead of compiling. Also the native code is stored in the cache, which is an expensive resource. With these circumstances, JIT compiler internally checks the frequency of each method call and decides to compile each only when the selected method has occurred more than a certain level of times. This idea of adaptive compiling has been used in Oracle Hotspot VMs.

Execution Engine qualifies to become a key subsystem when introducing performance optimizations by JVM vendors. Among such efforts, the following 4 components can largely improve its performance:

• Intermediate Code Generator produces intermediate code;
• Code Optimizer is responsible for optimizing the intermediate code generated;
• Target Code Generator is responsible for generating Native Code (i.e.Machine Code);
• Profiler is a special component, responsible for finding performance bottlenecks a.k.a.hotspots.

3.3. Garbage Collector

As long as an object is being referenced, the JVM considers it alive. Once an object is no longer referenced and therefore is not reachable by the application code, the garbage collector removes it and reclaims the unused memory.

4. Java Native Interface (JNI)

This interface is used to interact with Native Method Libraries. This enables JVM to call C/C++ libraries and to be called by C/C++ libraries which may be specific to hardware.

5. Native Method Libraries

This is a collection of C/C++ Native Libraries which is required for the Execution Engine and can be accessed through the provided Native Interface.

6. JVM Threads

The JVM concurrently runs multiple threads, some of these threads carry the programming logic and are created by the program (application threads), while the rest is created by JVM itself to undertake background tasks in the system (system threads).

The major application thread is the main thread which is created as part of invoking public static void main(String[]) and all other application threads are created by this main thread. Application threads perform tasks such as executing instructions starting with main() method, creating objects in Heap area if it finds new keyword in any method logic etc.

The major system threads are as follows:

• Compiler threads: At runtime, compilation of bytecode to native code is undertaken by these threads;
• GC threads: All the GC related activities are carried out by these threads;
• Periodic task thread: The timer events (i.e. interrupts) to schedule execution of periodic operations are performed by this thread;
• Signal dispatcher thread: This thread receives signals sent to the JVM process and handle them inside the JVM by calling the appropriate JVM methods;
• VM thread: This thread waits for operations to appear that require the JVM to reach a safe-point where modifications to the heap can not occur. The type of operations performed by this thread are “stop-the-world” garbage collections, thread stack dumps, thread suspension and biased locking revocation.

7. Conclusion

Java is considered as both compiled (high-level java code into bytecode) and interpreted (bytecode into native machine code). By design, Java is slow due to dynamic linking and run-time interpreting, however, JIT compiler compensate for the disadvantages of the interpreter for repeating operations by keeping a native code instead of bytecode.

8. Useful Commands

• javac: Java compiler;
• javap: Dump .class data;
• -XX:+PrintCompilation: Log every time a method is compiled to native code;
• -XX:+PrintInlining: Display a tree how methods has been inlined;
• -XX:+PrintAssembly: Look at the native code that JVM is outputting;
• jps: lists running Java processes;
• jcmd: used to send diagnostic command requests to the JVM
  • jcmd (without any parameters): list all JVM processes;
  • jcmd [PID] help: show available commands;
  • jcmd [PID] GC.heap_dump [PATH]: heap dump;
  • jcmd [PID] Thread.print: Thread dump.

9. Sources

• JVM Internals
• Understanding JVM Internals
• JVM Explained
• Java Virtual Machine Architecture in Java
• How JVM Works - JVM Architecture
• Java Virtual Machine (JVM) & its Architecture

Update Note: This RxJava series was written for RxJava 2.x (2019). RxJava 3.x introduced breaking changes and improvements. Core concepts remain the same, but some APIs have changed. Refer to the RxJava 3.x migration guide for differences.

Interested in Reactive Extensions and RxJava, I enjoyed reading the excellent book: Learning RxJava by Thomas Nield, and the following are my notes.

Why RxJava?

• Concurrency, event handling, obsolete data states, and exception recovery.
• Maintainable, reusable, and evolvable.
• Allows applications to be tactical and evolvable while maintaining stability in production.

Quickstart

In ReactiveX, the core type is the Observable which essentially pushes things. A given Observable<T> pushes things of type T through a series of operators until it arrives at an Observer that consumes the items. The following is an example of an Observable<String> that pushes three String objects:

fun main() {
    val observable = Observable.just("Hello", "world", "!")
}

Running this main method isn’t doing anything other than declare a Observable<String>. To make this Observable actually emit these three strings, an Observer need to subscribe to it and receive the items:

fun main() {
    val observable = Observable.just("Hello", "world", "!")
    observable.subscribe {
        print("$it ")
    }
}

This time, the output is the following:

Hello world! 

What happened here is that Observable<String> pushed each String object once at a time to the Observer lambda.

It’s possible to use several operators between Observable and Observer to transform each pushed item or manipulate them, the following is an example of map():

fun main() {
    val observable = Observable.just("Hello", "world", "!")
    observable.map { it.uppercase() }.subscribe { print("$it ") }
}

The output should be:

HELLO WORLD!

RxJava vs Java 8 streams

How Observable is any different from Java 8 Streams or Kotlin sequences? The key difference is that Observable pushes the items while Streams and sequences pull the items.

RxJava Series Guide

This is a comprehensive guide to RxJava organized by topic. Follow the links below for in-depth coverage:

Fundamentals

• Observable & Observer - Core concepts and Observable factories
• Hot vs Cold Observable - Understanding observable behavior patterns
• Observable Factories - Additional factory methods (range, interval, timer, etc.)
• Disposing - Resource management and stopping emissions

Operators

• Filtering & Control: Suppressing - filter, take, skip, distinct
• Transformation: Transforming - map, flatMap, concatMap, switchMap
• Aggregation: Reducing - count, reduce, all, any
• Collection: Collection - toList, toMap, collect
• Error Handling: Recovery - onErrorReturn, onErrorResumeNext
• Side Effects: Action - doOnNext, doOnComplete, doOnError

Advanced Topics

• Combining Observables - merge, concat, zip, combineLatest
• Multicasting - ConnectableObservable and sharing streams
• Replaying and Caching - replay() and cache() operators
• Subjects - PublishSubject, BehaviorSubject, and more
• Concurrency - subscribeOn and observeOn with Schedulers
• Parallelisation - Parallel execution strategies

Flow Control

• Buffering - Batch emissions into collections
• Windowing - Batch emissions into separate Observables
• Throttling - Control emission rate
• Switching - Cancel previous Observables

Backpressure

• Backpressure - Understanding and handling backpressure
• Flowable - Observable with backpressure support
• Subscriber - Consuming Flowables

Customization

• Transformers - Reusable operator chains
• Custom Operators - Building your own operators

Sources

• Learning RxJava
• ReactiveX Documentation
• RxJava Github
• RxMarbles

Note: code examples in this article are written in Kotlin to showcase the interoperability between Java and Kotlin, however, for Kotlin projects, it is most likely better to use RxKotlin.

Recently, I enjoyed reading a blog post by Jake Wharton about how Android supports Java 8 features using D8.

The blog post goes through the following processes to understand how D8 works:

1. Write Java code. (.java)
2. Compile to ByteCode.(.class)
3. Compile to Dalvik Executable. (.dex)
4. Analysis of the generated files.

In the blog post, the above process allows us to understand what happens under the hood when some Java 8 features (Lambdas and APIs) are desugared using D8.

In this post, we will use the same process to understand how default methods and static methods in Java 8 interfaces are desugared using D8. To better understand this post, I heavily recommend reading Jake Wharton’s post first.

Compile Java 8 Code

We will try to analyse the following code :

class Java8 {

  interface Logger {
    void log(String s);

    default void log(String tag, String s) {
      log(tag + ": " + s);
    }

    static Logger systemOut() {
      return System.out::println;
    }
  }

  public static void main(String... args) {
    sayHi(s -> System.out.println(s));
    Logger.systemOut().log("hello from static");
  }

  private static void sayHi(Logger logger) {
    logger.log("Hello!");
    logger.log("hello from", "default");
  }
}

We compile the java code:

$ javac *.java
$ ls
Java8.java  Java8.class  Java8$Logger.class

Executing the above code gives the following output:

$ java Java8
Hello!
hello from: default
hello from static

Then we compile the bytecode to dex using D8:

$ $ANDROID_HOME/build-tools/28.0.2/d8 --release --lib $ANDROID_HOME/platforms/android-28/android.jar --output . *.class
$ ls
Java8.java  Java8.class  Java8$Logger.class  classes.dex

Our focus here is the default and static methods in the Logger interface.

Dex Analysis

To see how D8 desugared interface’s static and default methods, we will use dexdump:

$ $ANDROID_HOME/build-tools/28.0.2/dexdump -d classes.dex

We get a lot of output (the full output can be found here).

Default Methods

Firs, we find the following output:

Class #0            -
  Class descriptor  : 'LJava8$Logger-CC;'
  Access flags      : 0x1011 (PUBLIC FINAL SYNTHETIC)
  Superclass        : 'Ljava/lang/Object;'
  Interfaces        -
  Static fields     -
  Instance fields   -

A new class Java8$Logger-CC has been generated! (We know it’s generated because of the SYNTHETIC flag). This class has Object as superclass and doesn’t implement any interfaces and have no static or instance fields.

Now let’s check these class methods. The class has two methods, the first one is $default$log:

Direct methods    -
    #0              : (in LJava8$Logger-CC;)
      name          : '$default$log'
      type          : '(LJava8$Logger;Ljava/lang/String;Ljava/lang/String;)V'
      access        : 0x0009 (PUBLIC STATIC)

We can read that this method is a static method and takes as arguments a Logger plus the same arguments as our default method in our Logger interface! The content of the method is:

[000434] Java8.Logger-CC.$default$log:(LJava8$Logger;Ljava/lang/String;Ljava/lang/String;)V
|0000: new-instance v0, Ljava/lang/StringBuilder; // type@000c
|0002: invoke-direct {v0}, Ljava/lang/StringBuilder;.<init>:()V // method@0012
|0005: invoke-virtual {v0, v2}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder; // method@0013
|0008: const-string v2, ": " // string@0001
|000a: invoke-virtual {v0, v2}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder; // method@0013
|000d: invoke-virtual {v0, v3}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder; // method@0013
|0010: invoke-virtual {v0}, Ljava/lang/StringBuilder;.toString:()Ljava/lang/String; // method@0014
|0013: move-result-object v2
|0014: invoke-interface {v1, v2}, LJava8$Logger;.log:(Ljava/lang/String;)V // method@0009
|0017: return-void

Even though the output looks complicated, the code here is actually simple and its logic is equivalent to the implementation of the default method in Logger interface! The equivalent Java code of the method can be the following :

public static void defaultLog(Logger logger, String tag, String s) {
    logger.log(tag + ":" + s);
}

We can conclude that the default method in interfaces are desugared to static methods in a newly generated utility class (Loger-CC).

Static Method

Based on what we already saw before, we can have a guess how static methods in interfaces are desugared! Let’s check! The generated class Loger-CC have a second static method! And without surprise, its name is systemOut:

#1            : (in LJava8$Logger-CC;)
  name        : 'systemOut'
  type        : '()LJava8$Logger;'
  access      : 0x0009 (PUBLIC STATIC)

The method systemOut in Logger-CC take no arguments and returns a Logger ! The body of the method is:

|[00040c] Java8.Logger-CC.systemOut:()LJava8$Logger;
|0000: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream; // field@0002
|0002: invoke-virtual {v0}, Ljava/lang/Object;.getClass:()Ljava/lang/Class; // method@0011
|0005: new-instance v1, L-$$Lambda$teOjDu261Kz9uXGt1wlPvIP5S04; // type@0001
|0007: invoke-direct {v1, v0}, L-$$Lambda$teOjDu261Kz9uXGt1wlPvIP5S04;.<init>:(Ljava/io/PrintStream;)V // method@0004
|000a: return-object v1

Without surprise, the code is equivalent to our implementation of the static method in systemOut in the interface Logger ($Lambda$teOjDu261Kz9uXGt1wlPvIP5S04 is a class that corresponds to the lambda System.out::println in our implementation).

The equivalent java code can be:

public static Logger systemOut() {
    return new Lambda(System.out); //System.out::println
}

Conclusion

The following Java code is an equivalent of our Java8 class above (PS: I changed some methods/classes names for clarity) :

import java.io.PrintStream;

class Java8Desugared {

  interface Logger {
    void log(String s);
    void log(String tag, String s);
  }

  public static final class LoggerCC {

    public static void defaultLog(Logger logger, String tag, String s) {
      logger.log(tag + ":" + s);
    }

    public static Logger systemOut() {
      return new LambdaSystemOut(System.out);
    }
  }

  public static final class LambdaSystemOut implements Logger {
    private PrintStream ps;

    public LambdaSystemOut(PrintStream ps) {
      this.ps = ps;
    }

    @Override public void log(String s) {
      ps.println(s);
    }

    @Override public void log(String tag, String s) {
      LoggerCC.defaultLog(this, tag, s);
    }
  }

  public static void main(String... args) {
    sayHi(LambdaSayHi.INSTANCE);
    LoggerCC.systemOut().log("hello from static");
  }

  private static void sayHi(Logger logger) {
    logger.log("Hello!");
    logger.log("hello from", "default");
  }

  public static final class LambdaSayHi implements Logger {
    static final LambdaSayHi INSTANCE = new LambdaSayHi();

    private LambdaSayHi() {}

    @Override public void log(String s) {
      lambdaContent(s);
    }

    @Override public void log(String tag, String s) {
      LoggerCC.defaultLog(this, tag, s);
    }
  }

  static void lambdaContent(String s) {
    System.out.println(s);
  }
}

Compiling and running the above code gives the same output as our Java8 class:

$ javac Java8Desugared.java
$ java Java8Desugared
Hello!
hello from:default
hello from static

Sources

• Android’s Java 8 Support
• Dalvik bytecode

#include <stdio.h>

int main(void)
{
        printf("hello, world\n");
        return 0;
}

For a long time, I had the idea of setting up my own website/blog, I’ve done some shy attempts before but this one is the one! So welcome to my blog, and to this unavoidable “Hello, world!” post! :D

I’ve built this blog with Jekyll using the Indigo theme, the code source is available at Github.

Cheers,
Mouaad