Tinkering

Yes, this post is going to be about Flox, my employer, but this isn't an advertisement. This is me, both an engineer and a user, describing why I think a new feature I worked on is interesting, how it can change the way you develop your software, and what it was like to take point on the feature. We called this feature "composition", and it allows you to build developer environments in a modular way out of other developer environments. It was initially released in Flox 1.4.0, and rounded out in 1.4.1. It's really cool. # The featureSVG Image ## Flox environmentsSVG Image The central abstraction in Flox is the "environment", which is a generalization of the "developer environment" since Flox environments are useful for more than just local development. For example, you can use a Flox environment situated in your home directory (we treat this specially and call it the default environment) to use Flox as a system-wide package manager, replacing brew or whatever your package manager of choice is. There are no containers or VMs involved. When you "activate" an environment you're placed into a new subshell (by default) whose environment is configured very carefully. Once you're inside the environment, you have access to the packages, environment variables, services, etc that you've defined as part of your environment. Environments also have configurable startup scripts that run when activating the environment. If you want to stay in your current shell rather than entering a new subshell, there are ways to make Flox spit out shell code that you can then eval. In fact, this is how you configure your shell to activate your default environment for every new shell: bash eval "$(flox activate -d ~ -m run)" Flox uses Nix under the hood for reproducibility (no, we aren't just calling nix develop or nix shell under the hood1), which means that we can enumerate and lock all of the packages, environment variables, scripts, etc that go into an environment, store that in a lockfile, and reproducibly build it on another machine. ## Combining environmentsSVG Image Since Flox environments are just shells, you can nest them. This allows you to "layer" environments. This also that means your developer environment can context switch with you. Say you work on a web service and you have an environment stored in git along with your source code. When you start work you cd myrepo followed by flox activate. Now you're ready to work. You discover a networking bug, and want to investigate further, so you flox activate --remote your_user/net_tools, an environment you've pushed to FloxHub that contains some extra network debugging tools. You're now in a nested subshell with access to both your development tools, and your debugging tools. Once you're done debugging, you exit and you're back to the shell that just has your development tools. The first time you do that and see how easy it is, it's a "hell yeah" moment. This makes context switching pretty painless, and it makes it possible to build up a tech stack from building blocks. In this case you're combining environments by layering them in succession. Let's say you have separate environments for your development tools (dev), running a Postgres server (postgres), and running a Caddy server as a remote proxy (caddy). In order to get access to all of these at the same time, you need to flox activate dev, flox activate postgres, and flox activate caddy every time you want to do work. There are some other drawbacks here, like a tool provided by the caddy environment shadowing one from the dev environment because it appears earlier in PATH (it was activated later). This makes layering suitable for ad-hoc tasks, but less well suited for building up a developer environment from building-block environments. That's where composition comes in, and it's so cool that it gives me nerd-glee that it exists. ## CompositionSVG Image The idea behind the composition feature is that you can merge environments rather than layering them. A high-level documentation page and a tutorial are available in the official documentation (both written by me, feedback welcome). Composing environments is trivially easy. You add the following section to your manifest.toml (our config file), listing the environments you want to merge: toml [include] # Later entries are given higher priority during the merge environments = [ # An environment present at a relative path { dir = "path/to/env" }, # An environment on FloxHub containing a Rust toolchain { remote = "zmitchell/rust" } ] This is so cool. You can now prepare independent developer environments for different contexts, and piece them together to cover the majority of your needs for new projects. Installing Python and your Python package manager of choice isn't the interesting part of working on a project, it's the stuff that makes the project unique. Define a Python environment that contains the interpreter, package manager, etc that you can bring to every new Python project, then focus on unique parts. Consider this scenario: you're working on a web service that depends on a database and a Rust toolchain to develop the service itself. Here's what all of that looks like: toml [include] environments = [ { remote = "zmitchell/rust" }, { remote = "zmitchell/postgres" } ] The other day one of my coworkers put together a Kafka environment and a Karapace environment (it's a Kafka schema registry thing, idk, I'm not a Kafka person), which you can then configure with environment variables: toml version = 1 [vars] KAFKA_MODE = "kraft-combined" KAFKA_NODE_ID = "1" KAFKA_HOST = "172.30.38.117" KAFKA_PORT = "9092" KRAFT_CONTROLLER_PORT = "9093" KAFKA_CLUSTER_ID = "EBzt0KoZR5ynZ9hTiJQuFA" KAFKA_REPLICATION_FACTOR = "1" KAFKA_NUM_PARTITIONS = "1" KAFKA_HEAP_OPTS = "-Xmx512M -Xms512M" REGISTRY_HOST = "172.30.38.117" REGISTRY_PORT = "8081" REST_HOST = "172.30.38.117" REST_PORT = "8082" LOG_LEVEL = "INFO" [include] environments = [ { remote = "barstoolbluz/kafka-basic-patch" }, { remote = "barstoolbluz/karapace-basic-patch" } ] ### How it worksSVG Image When the environment is built, the manifests of all of the "included" environments will be merged, and then the manifest of the "composing" environment will be merged into that. This creates a "merged" manifest from which we lock and build a "composed" environment. You don't edit this merged manifest directly, but you can surface it via flox list -c 2, which will print out the merged manifest if one exists, or the normal manifest if a merged manifest doesn't exist. In case of conflicts between data contained in different manifests, entries later in the include.environments array are given higher priority, and the composing environment's manifest takes the highest priority. This allows you to fix situations where one environment shadows a package/variable provided by another: the composing environment can redeclare the package/variable to override and fix the situation. As part of the design process we discussed more granular overrides, but we decided to keep things simple for this first pass. The included environments will likely see updates over time, so we also provide the flox include upgrade command to pull in the latest manifests, re-merge, and build the environment. Ok, nifty, let's talk about how it affects your workflow. ## My new workflowSVG Image I have ADHD, which means I start new projects more often than most people, which means that I probably feel the pain of setting up developer environments for new projects more often than most people. My development workflow has changed over the years, but it has gone through these rough phases: YOLO, Nix, and now Flox. The YOLO workflow was basically Rust installed via rustup, Python installed via pyenv installed via brew, and then random other dependencies installed through brew as well. I used containers at work, but always found them kind of a pain in the ass for side projects between needing to SSH into them, mount in directories, expose ports, deal with inevitable file permissions issues because I fucked something up, etc. So, while I acknowledge that containers exist and work for a lot of people, I skipped that phase for personal projects and went straight to Nix. The Nix workflow was basically copying and pasting flake templates from one project to the next. That was also kind of a pain in the ass, but it was a one-time upfront pain in the ass when setting up a new project, and then everything tended to just work. Then, once Flox got to a point that I was comfortable using it for my local development, I bootstrapped new projects with one command: flox pull --copy zmitchell/rust. This gives me a copy of my Rust toolchain environment that I can then extend with project-specific dependencies, disconnected from the copy that's stored on FloxHub. That's how I've worked for a while now, and it works well. There are no pains in my ass with this workflow, but it does mean that I have a handful of projects using essentially the same tools but with no connection to one another. Furthermore, since these projects are disconnected from each other, it's possible that I have multiple copies of very similar toolchains in my Nix store, taking up space. This also means that if I add one package to my Rust toolchain (e.g. pkg-config), none of these disconnected projects will get that update. With composition I could add pkg-config to zmitchell/rust once, and then I'll get it the next time I flox include upgrade an environment that includes zmitchell/rust. This also means that if I go flox include upgrade any environments that use zmitchell/rust, I'll only have one copy of that toolchain on my system rather than N. If I want to pull updates to one of the enviroments (e.g. zmitchell/rust), but not the others, I can do that too. Currently, with composition, my initial setup is slightly longer, but it's cleaner and let's me retain the connection to and history of the original environment on FloxHub. Now I flox init, followed by flox edit to add this: toml [include] environments = [ { remote = "zmitchell/rust" } ] With composition, my manifest contains the list of environments that I want to build off of, and then only the very specific things I want for this project. That keeps the manifest.toml concise, and allows for a separation of concerns not really possible with other tools. We have lots of ideas for how to make this workflow better. For instance, I have an open ticket for creating a flox include add command (or something like it) that will let me include zmitchell/rust from the command line rather than editing it in manually. We also want to add the ability to pin remote environments, but there's design work to be done before we implement that. ## Why this mattersSVG Image The reason this is important is that it's now possible and easy to build developer environments out of reusable, composable building blocks. Think of how much time and effort that saves! If I set up a Rust toolchain, a Python toolchain, etc why should I ever do that same work again? Just include it and add in your project specifics. This isn't the main use case for me, but we've also already used this feature to provide support to some of our users e.g. they want a feature that doesn't exist yet, but we give them an environment to include whose setup script does the thing they're asking for. My coworkers and I all had the same experience using this for the first time. We all tried it out, and it just worked, and we said "hell yeah." # What it was like to work on itSVG Image So, with all of that out of the way, let's get a little bit meta and reflect on what it was like to work on it. ## Work planSVG Image I've often wished that projects had something like a directed graph of what the work looks like so that I can see how much there is in total, how much has been completed, how much is remaining, where decisions will block implementation, etc. Since I was the lead for this feature, I decided to make one in Figma. I tried to color code "sticky notes" by their completion status and whether they were implementation work or decisions. No one else used this (it was mainly for me), but it did help defer certain decisions in favor of getting started on the engineering work, which was helpful for getting it out the door faster. To be clear, the actual work was split out into workable GitHub issues, but the graph helped keep an eye on the high level progress and direction. ## Design decisionsSVG Image For this first pass the user experience isn't perfect, but I think it laid the ground work well. The main design choice was whether to lock and record the manifests of the included environments at merge-time. If you lock and record the manifests you can re-merge at any time without needing to fetch those manifests again, allowing you to decouple "install a new package" from "get me the latest manifests". If you don't lock and record the manifests, you're required to fetch the latest manifests any time you build the environment, which gives you an auto-fetch-and-upgrade mechanism out of the box. I argued for the lock-and-record case because the other case allows you to get in weird situations (I'll omit the details for the sake of time), but it does mean that some things you wish were automatic are currently manual (e.g. you need to run flox include upgrade to get updates to included environment manifests). That said, the ground work is laid for an intentional auto-upgrade mechanism. The next big design question was how to implement merging the manifests. We considered two main options. Our manifest is currently represented with a strongly-typed Manifest struct (the CLI is written in Rust), so one option was to manually write out the traversal of the manifest struct, with all the friction that entails due to the nested structure and strong typing. The other option was to treat the TOML like JSON, and merge serde_json::Value structs (this is a generic JSON object, for you non-Rust people). Eventually we landed on the manual, strongly-typed option because some of the manifest fields make more sense to overwrite during the merge rather than strictly merging. One example of this is the list of command line arguments to use as the Cmd when you bundle up a Flox environment into a container (merging ["bash", "foo"] with ["bash", "bar"] to create ["bash", "foo", "bash", "bar"] is probably not what a user expects). Looking ahead to some future work where we want to provide structured information about the diff between two manifests, it may be worth creating a procedural macro that can automatically generate a visitor pattern trait for comparing two manifests. That would look something like this: rust trait ManifestCompareVisitor { fn compare_install(&mut self, install_left: &Install, install_right: &Install); fn compare_vars(&mut self, vars_left: &Vars, vars_right: &Vars); ... } This would allow us to have one interface that can be used for both merging and diffing. # ConclusionSVG Image If you've made it this far, thanks for reading, no one has ever accused me of having too little to say. Give Flox a try, let me know what you think about this composition feature, etc. Is it interesting to hear about what the work process is like? I've never really written about that before, but found it kind of cathartic. 1 We basically wrote our shell-handling code from scratch, and I have the scars to prove it. If you use Flox with Zsh, you owe me and my team a collective drink. I think 10 different lifecycle files ({~, /etc/}.{zshenv, zlogin, zlogout, zprofile, zshrc}) is a bit much. 2 It's one of my pet peeves that flox list --config (1) doesn't list anything, it prints one thing, (2) the thing we're printing is the "manifest", not the "config", and (3) we have a separate thing called "config" that this command doesn't print. Humbug.

At Flox1 we recently released a feature that we call "service management", which is essentially an analog to Docker Compose that runs processes instead of containers. In short, you define some services in your manifest and they get started as part of activating your environment with flox activate --start-services. When there are no more activations of this environment (i.e. you've closed all your shells that had activated the environment) the services are automatically cleaned up. A simplified environment that starts a Postgres server looks like this: toml version = 1 [install] postgres.pkg-path = "postgres" [services.postgres] command = "postgres -D my_data_dir" [options] systems = ["aarch64-darwin", "x86_64-darwin", "aarch64-linux", "x86_64-linux"] Pretty straightforward, right? What's not straightforward is wrangling a process manager to bend it to your will without losing your mind a little bit. What's the issue? Race conditions. Race conditions everywhere. Some of them in how you call the process manager, others inside the process manager itself. Different process managers will be better or worse at certain tasks, and some will be missing features that you've decided that you really need. They may also just do different things because there's not a single correct answer. As always, choosing one will come down to your particular needs and priorities. However, the unknown unknowns can make choosing pretty difficult. Having spent the last couple of months immersed in this, I've put together a list of topics to consider to help you get your due diligence done. # ShutdownSVG Image - Does it cleanly shut down when requested (e.g. with a built-in command)? - Does it cleanly shut down when sent a SIGTERM, etc? - If there is data in-flight (process has shut down, but there are logs yet to be written) during shutdown, is shutdown postponed until that data is persisted, or is it gone forever? # Starting servicesSVG Image - Can you start a single process? - Can you start all processes without naming them? - Can you specify a startup order? - Can it block until processes have started? - Is it clear what it means for processes to have started (e.g. the process has been forked vs a readiness check is green)? - What happens if you try to start a process that's already running (warning? error? success?) - What happens if you try to start multiple processes and some of them are already running? Do you get individual warnings but still succeed? Do you get an error? - What happens if you try to start multiple processes and some of them don't exist? Is that a warning, a failure, or success? Do you get individual warnings/errors? Does it check the names beforehand? Does it succeed until it finds a name that doesn't exist? # Stopping processesSVG Image - Can you stop a single process? - Can you stop all processes without naming them? - Can you specify a shut down order? - Can it block until all processes have stopped? - Is it clear what it means for processes to have stopped (e.g. the process has been sent a SIGTERM vs the process has terminated and is now a zombie vs the process has terminated and been cleaned up)? - What happens if you try to stop a process that's not running (warning? error? success?) - What happens if you try to stop multiple processes and some of them aren't already running? Do you get individual warnings and succeed? Do you get an error? - Does it distinguish between processes that aren't running and processes that don't exist? - What happens if you try to stop multiple processes and some of them don't exist? Is that a warning, a failure, or success? Do you get individual warnings/errors? Does it check the names beforehand? Does it succeed until it finds a name that doesn't exist? # Restarting processesSVG Image - Can you restart a single process? - What about dependent processes in the startup order? Shutdown order? - Can you restart all processes without naming them? - Can it block until processes have restarted? - What happens if you try to restart a process that's isn't running (warning? error? starts it?) - What happens if you try to restart multiple processes and some of them aren't already running? Do you get individual warnings and succeed? Do you get an error? - What happens if you try to restart multiple processes and some of them don't exist? Is that a warning, a failure, or success? Do you get individual warnings/errors? Does it check the names beforehand? Does it succeed until it finds a name that doesn't exist? # BackgroundingSVG Image - Does it run in the foreground or background by default? - If it runs in the foreground by default, does it have a way to background it, or do you have to do it manually? # Client/server (assumes backgrounding)SVG Image - Does the server daemonize (re-parent) itself? - Does it use a Unix socket or TCP? - If you'll have more than one running, how do you prevent conflicts (ports, socket location, etc). - If it uses a Unix socket, can you configure or predict its location? - If a dead socket file is found on startup, is that an error or does it clean it up and create a new one? - Can multiple clients connect to the same server at the same time? Does that affect data integrity i.e. does each client get a complete set of streaming logs? - If it shuts down does it wait until all outstanding responses are sent before terminating the server? - Can you tell whether the server is running without sending it a command? # LogsSVG Image - Are the process manager logs separate from the process logs? - Are the process logs separate from each other, or in one log stream? - Can the log format be configured (e.g. human readable, json, etc) - Can the process manager logs be persisted or only printed to the terminal? - Can the process logs be persisted or only printed to the terminal? - Can you stream the logs? Can you stream all processes? Can you stream a single process? Can you stream some but not all processes? - Can you sample the last few log lines? For all processes? For one? For some but not all? # UpdatesSVG Image - Can you add or remove processes without restarting the process manager? - If there's a startup order, does it restart any services when the startup graph changes? # StatusesSVG Image - What information are you given about the processes? Can you see the runtime? PID? Number of restarts? Name? - Is the list of statuses granular enough? Too granular? Is it clear what the differences are between the statuses? - What is the status of a process that has never been started? That crashed? That completed? That was terminated via signal? Can you tell what signal? Can you see the exit code? 1 At Flox we're kind of shaking the Etch-A-Sketch on developer environments. Cross-platform, reproducible developer environments without containers so you get the best of both worlds: your tools and dependencies are the same from engineer to engineer, machine to machine, and you get to keep all the tools you love, configured the way you want. But, that's not the point of this post.

# Ghost triggersSVG Image This particular experiment was driven by a laser that periodically spit out pulses that were 5ns long. A photodiode would pick up when a pulse was emitted, and we'd use that as the trigger for the oscilloscope. The oscilloscope would then record signals from our two other detectors. One of these is called "probe" and the other is called "reference". Here's a block diagram of the system: Only the beam hitting the probe detector actually passes through the sample. The beam hitting the reference detector does not pass through the sample, so any variations in intensity must come from noise in the laser, vibrations in the table, etc. All of these variations should be seen in both the probe and reference beams since they were produced by splitting a "parent" beam. We do this so that if you divide the probe signal by the reference signal you cancel out any part of the signal that doesn't come from the physics happening inside the sample. This isn't perfect, but it works well enough. Imagine you need to run this experiment for about 8 hours in order to get enough measurements. Now imagine the oscilloscope is running Windows XP with the tiniest hard drive known to man. If you're not a programmer and you don't know that you can drive the experiment from Python, and don't know that you can save every measurement individually, it makes sense that you'd resort to averaging the data in place on the oscilloscope. Again, this isn't perfect, but it works well enough. That is, until one day when you start mysteriously averaging in a garbage measurement once in a while, destroying all the data you've just collected. It looked as if the running average was trying to incorporate a measurement that was either 0/<very small number> or <anything>/0. This wasted hours of time when it happend, but at least the squiggly line on the screen looked cool. We try all the obvious things, then we all give up and the new guys in the lab (me and someone else) decide to just sit and watch the experiment run in hopes of witnessing the failure. We wait, and we wait. No failure. Until someone unplugs their phone charger from the wall. The fuck? We plug the charger back in and unplug it, just to make sure that the insane thing we're seeing is reproducibly insane. Yup, it's reproducibly insane. It turned out that a large amount of electrical interference was emitted every time a phone charger was unplugged from the wall (although not from Apple iPhone chargers, take from that what you will). Electrical interference is always floating through the air, like it or not, but electrical engineers and physicists (mostly) know about this. To protect our equipment from interference we "shield" it, which really just means we wrap it in metal. Guess what, the photodiode we were using to pick up the laser pulse wasn't shielded very well. One might even say that the shielding was absolute dogshit. Couple that with a very old, very crappy cable connecting it to the oscillscope and you have yourself what is basically an antenna. Whenever this photodiode or cable would pick up interference, it would like to the oscilloscope as if a laser pulse had just arrived. It would then trigger a measurement, even though there was no laser beam present. This is where the 0/<very small number> or <anything>/0 came from. # Ok, I guess I have to worry about that nowSVG Image I was working on an experiment that used a never-before-seen measurement technique. The laser system itself was incredibly sensitive and used 20fs pulses (a femtosecond is 10^-15 seconds). I'm pretty sure it was the most sensitive device of its kind in the world by 1 or 2 orders of magnitude. I was pretty proud of it. However, when you have a device that's incredibly sensitive, you start to pick up effects that are well below the noise floor of other measurement techniques. This will become relevant. Here's a diagram of the system from my thesis: A few things to point out here: - Yes, I did make this diagram in LaTeX. - Yes, of course there are LaTeX packages for drawing optical layouts. - Yes, I am insane. - Yes, I'm one of the only people to ask about how to draw an off-axis parabolic mirror in LaTeX. - This diagram is meant for physicists, the only thing you need to see is that there's a 950kHz "AOM" and a 50kHZ "PEM". The way this experiment works is that you have two laser beams that intersect in the sample. One of them is called the "probe", and one of them is called the "pump". In short, the pump excites the sample, and the probe comes by some time later to see how things have evolved after some period of time. In this case we're modulating each beam at some frequency, 50kHz for the probe (modulating the polarization), 950kHz for the pump (modulating the intensity). Why? Well, when these beams interact in the sample, the physics we're trying to observe happens at the sum of these two frequencies, which in this case is 1MHz. We use a special instrument called a "lock-in amplifier" to only detect a signal at 1MHz, which eleminates all the noise present at other frequencies. It's incredibly effective...when you don't have other sources noise at 1MHz. I had to put together a rats nest of electronics to make sure that my two signals, 50kHZ and 950kHz, never got mixed together except for the exact place that I wanted them mixed. You can see that box here: Note the "LOL" spelled out in gaffer tape. I put that there for whoever the next student was because I intended to never open this box again, and good luck to whoever had to open this box and play with it. The joke was on me, clearly. Again, this device is incredibly sensitive, and one day I'm completely stuck trying to figure out where a source of 1MHz noise was coming from. My electronics are isolated extremely well. My lasers are modulated extremely well. There's no 1MHz electrical interference in the lab. What the hell is going on? I mentioned above that we used two devices, an "AOM" and a "PEM", to modulate each beam. An "AOM" is an "acousto-optic modulator", and it modulates the intensity of a laser passing through it. It takes an incoming electric signal (my 950kHz signal, in this case) and uses it to essentially vibrate a piece of glass in a special way. A "PEM" is a "photo-elastic modulator", and it modulates the polarizating of a laser passing through it. It also takes an incoming electrical signal (my 50kHz signal, in this case) and uses it to essentially vibrate a piece of glass in different but also special way. You'll notice that these devices both work by vibrating glass. Long story short, it turned out that ultrasonic vibrations from the PEM were floating through the air in the system and vibrating the glass in the AOM. The 50kHz ultrasonic vibrations picked up in the AOM were mixed together with the 950kHz vibrations that the AOM was correctly producing, producing a 1MHz signal. I was absolutely floored. Normally when you work with lasers you don't have to worry about sound. Like, ever. Lasers are light, they don't typically care about sound. Everyone who has done a PhD has thing kind of moment where you think to yourself, "sure, I'll just add this to the pile of bullshit that I didn't know I needed to worry about." The solution was to simply put a glass plate on either side of the PEM in order to contain the ultrasonic vibrations it was producing. You can see that in this photo, also held together with gaffer tape: # My shirt is the bugSVG Image I was working on another experiment that used yet another very sensitive measurement technique. The photodiode I was using in this case was tuned to have very good time resolution and frequency response under low-light conditions. This photodiode and this measurement technique were both very sensitive to stray light, so we built a little box out of plexiglass and covered it in matte-black aluminum foil (yes, that exists) to block out all of the stray light. The back of the box wasn't made out of plexiglass. This was necessary for practical reasons, namely that I frequently needed to get inside the box and tweak the optics around the photodiode. To cover the back of the box during an experiment we simply draped a black blanket over it. This worked shockingly well. At some point I'm debugging an issue with the system. I notice that there's a huge spike in the data right around when a laser pulse arrives, but this is a separate laser than what the photodiode is supposed to detect, and it's not pointed at the photodiode. This tells me that somehow all the mechanisms I'm using to protect the photodiode against stray light aren't quite working. I spend a lot of time working on this. At one point I'm running the system in real time, and I'm watching the signals on the oscilloscope so I can see how the system responds when I block light in various places. Then I notice that this "bad" signal changes depending on where I'm standing. What. the. fuck. I start to have PTSD flashbacks to the "lasers vs. sound" incident. I question my life choices. How can this be possible? Is there some kind of radio frequency interference that's blocked when I stand next to the experiment? Let's talk about where stray light comes from. Well, one source is the overhead lights. Another source is the air. Yup. The lasers we're using in this experiment are very, very intense, and as the light travels through the air some amount of it will literally bounce off of the molecules that make up the air, sending that light in a completely random direction. That light continues to bounce off whatever else is in the room. Like me. It turned out that there was two issues. I had already solved the real issue. The other issue, the one that seemed to depend on where I stood, was only present when I was debugging the first issue. It so happens that this blanket that was draped over the back of the box during experiments was pulled up slightly while I was diagnosing the real issue. And since it was pulled up, stray light could get in the back of the box. Stray light like the random photons bouncing off of molecules in the air, which then bounce around like pinballs until they eventually bounce off my shirt and into the back of the box, even though this beam is originally pointed in the opposite direction. Sigh.

At $WORK I have a need to create some bitmasks over a 64-bit field. I've never done this before. I figured that surely tables exist for grabbing a certain number of bits from a certain position, but I never found those tables. So, to scratch my own itch I wrote a very simple Python program to generate the tables and used a CSV to Markdown converter to create the tables. Here you go, now we both have a quick reference. Update 2023-04-04: Someone on Mastodon mentioned bitwise, an interactive terminal program for bit manipulations. It looks pretty cool! ## ProgramSVG Image python bit_sizes = [1, 8, 16, 32] total_bits = 64 for bit_size in bit_sizes: positions = int(total_bits / bit_size) n = 0 for _ in range(bit_size): n = n << 1 n += 1 print(f"# {bit_size} bits") for pos in range(positions): print(f"{pos}, {n}") n = n << bit_size ## Table formatSVG Image Each table has a "position" column and a "decimal value" column. The "position" refers to which n bits you're setting to 1. For a single bit the position is which individual bit is 1. For 8 bits the position is which 8 bits you're setting to 1. The decimal value is the decimal value when the bits at the given position are set to 1. For example, 8 bits in position 1 (the second byte) corresponds to the binary number 0b000....01111111100000000 and the decimal value 65280. To get the mask for an odd number of bits you can just add the decimal values in the table since the positions don't overlap for a given number of bits. ## 1 bitSVG Image | Position | Decimal Value | | -------- | ------------------- | | 0 | 1 | | 1 | 2 | | 2 | 4 | | 3 | 8 | | 4 | 16 | | 5 | 32 | | 6 | 64 | | 7 | 128 | | 8 | 256 | | 9 | 512 | | 10 | 1024 | | 11 | 2048 | | 12 | 4096 | | 13 | 8192 | | 14 | 16384 | | 15 | 32768 | | 16 | 65536 | | 17 | 131072 | | 18 | 262144 | | 19 | 524288 | | 20 | 1048576 | | 21 | 2097152 | | 22 | 4194304 | | 23 | 8388608 | | 24 | 16777216 | | 25 | 33554432 | | 26 | 67108864 | | 27 | 134217728 | | 28 | 268435456 | | 29 | 536870912 | | 30 | 1073741824 | | 31 | 2147483648 | | 32 | 4294967296 | | 33 | 8589934592 | | 34 | 17179869184 | | 35 | 34359738368 | | 36 | 68719476736 | | 37 | 137438953472 | | 38 | 274877906944 | | 39 | 549755813888 | | 40 | 1099511627776 | | 41 | 2199023255552 | | 42 | 4398046511104 | | 43 | 8796093022208 | | 44 | 17592186044416 | | 45 | 35184372088832 | | 46 | 70368744177664 | | 47 | 140737488355328 | | 48 | 281474976710656 | | 49 | 562949953421312 | | 50 | 1125899906842624 | | 51 | 2251799813685248 | | 52 | 4503599627370496 | | 53 | 9007199254740992 | | 54 | 18014398509481984 | | 55 | 36028797018963968 | | 56 | 72057594037927936 | | 57 | 144115188075855872 | | 58 | 288230376151711744 | | 59 | 576460752303423488 | | 60 | 1152921504606846976 | | 61 | 2305843009213693952 | | 62 | 4611686018427387904 | | 63 | 9223372036854775808 | ## 8 BitsSVG Image | Position | Decimal Value | | -------- | -------------------- | | 0 | 255 | | 1 | 65280 | | 2 | 16711680 | | 3 | 4278190080 | | 4 | 1095216660480 | | 5 | 280375465082880 | | 6 | 71776119061217280 | | 7 | 18374686479671623680 | ## 16 bitsSVG Image | Position | Decimal Value | | -------- | -------------------- | | 0 | 65535 | | 1 | 4294901760 | | 2 | 281470681743360 | | 3 | 18446462598732840960 | ## 32 bitsSVG Image | Position | Decimal Value | | -------- | -------------------- | | 0 | 4294967295 | | 1 | 18446744069414584320 |