1. TCP gives you a stream of bytes, not packets.
2. read() can give you fewer bytes than you asked for.

If either of those was a surprise: keep reading.

Byte streams

The main misconception people have is that when they're reading from a TCP socket, they are receiving packets. This is the wrong way to think about it. If you're writing anything higher level than the TCP implementation itself, then you should forget about packets. TCP is exposed to you via a pair of byte streams.

Typically both streams are on the same file handle (they have the same file descriptor), but remember there are two underlying streams: writing puts bytes into the stream that is sent to the other side, and reading gets bytes out of the stream coming from the other side.

There are a few reasons that people are not immediately disabused of packet-oriented thinking:

• When you learn about networking, you learn about packets, so it is natural to assume that you'll be handling packets.
• When you write test programs to send short strings over localhost, it probably appears as if every write() pops out in a corresponding read().
• When you test your server with netcat, it appears as if each line is sent as a discrete "packet" because the line-buffering on netcat's stdin means each line doesn't get read by netcat until you press enter, and then it all gets sent to the socket at once.

If you write a program that tries to read "packets", there are a handful of potential issues you can encounter:

• A single read might get less than an entire "packet", your program may think it is a malformed packet, and you'll drop some data and potentially get out of sync with the stream.
• A single read might get more than an entire "packet", your program may not notice the extra, and you'll drop some data and potentially get out of sync with the stream.

Just stop trying to think about packets. The kernel will deal with packets. You will deal with byte streams.

To transfer discrete messages over a byte stream, you need some sort of message encoding. Some simple schemes include:

• Fixed length: every message is the same length, so you know you have a full message when you have that many bytes (see Protohackers problem 2).
• Predictable length: not every message is the same length, but you can compute how long the message is going to be based on what type of message it is (see Protohackers problem 6).
• Length prefixed: the start of each message tells you how many bytes the message is.
• Line delimited: each message is a line of text, so messages are terminated by newline characters (see Protohackers problem 1).

You can do whatever you want as long as you can work out where the message boundaries are. If you want to transfer structured data I suggest JSON lines for a text format or length-prefixed protobuf for a binary format.

Read semantics

The other misconception is about the semantics of reading from a socket.

Depending on the platform you are using, this could bite you in 2 different ways. Normally read() takes an argument saying how many bytes you want, and then there are 2 common ways for it to work:

1. it gives you back any amount of bytes, up to the maximum you gave
2. it blocks, and keeps reading, until either the end of the stream, or it gives back exactly the number of bytes you asked for

The actual read() system call is "type 1": if there are some bytes available immediately, but not as many as you asked for, you'll just get back whatever is available immediately.

C's fread() works the second way: it blocks until either the end of the stream, or it has exactly the number you asked for.

Neither of these semantics is necessarily "better" than the other. In the first case, you have to manually make sure you have all the bytes you want (i.e. keep calling read() until you have enough). In the second case, you have to make sure you don't block the entire program in the course of getting all the bytes you want (e.g. if you want 5 bytes but the kernel only has 1 to give you, fread() will block even though select() told you the socket was readable).

The following properties are common to both types:

• it can give you less than you asked for (in the case of "type 2": only at the end of the stream)
• it can block indefinitely if no data is available (in the case of "type 2": it can also block indefinitely even if some data is available!)
• it can give you less than one full packet's worth of data (but you don't care about packets!)
• it can give you more than one full packet's worth of data (but you don't care about packets!)
• it is not the case that one call to write() on the other side will exactly land in one call to read() on your side

Here are some classifications that I'm aware of:

• C: read(): type 1, fread(): type 2
• Perl: read(): type 2, sysread(): type 1
• C#, NetworkStream: Read(): type 1, ReadExactly(): type 2
• Go, io.Reader: Read(): type 1, ReadFull(): type 2

If you want to write reliable software, you need to find out what read semantics you're using. Don't just guess.

(Also, for what it's worth, the write() system call has the same property as read(), in that it may write fewer bytes than you asked it to; check the return value to know how many bytes were actually written, and then try again to send the rest).

Half-close

While you're here, I have an axe to grind...

There's one more thing that I want you to know: half-close. You remember how we agreed that there are 2 separate byte streams? Well, corollary to that is that the 2 streams can be closed independently: you can close the stream you are writing to, even while you still want to read from the other side. If you understand the stream abstraction, this should be natural and good.

Sadly, some "transparent proxies" inadvertently break half-close, by tearing down connections as soon as they see either of the streams get closed. Please don't do this!

The correct thing to do is to propagate the half-close onwards. If you imagine the TCP session as a pipeline running from the client, through the proxy, to the server, and then back through the proxy to the client, then it is obvious that the half-close should be propagated on through the pipeline in the same path that normal messages would take. A half-close shouldn't "jump ahead" of any pending data in the pipeline by closing the session immediately.

(I wanted to draw a neat animation showing a client, a proxy, and a server, with water pipes connecting them, and in the good case, when the flow from the client to the proxy is shut off, all the valves will get shut off one by one, in order, through the whole pipeline, following the last of the flowing water; and in the bad case, as soon as the first valve is shut off, the rest of the pipes all get closed at once and the water they contain is dumped on the ground. But making animations is too much trouble, so please imagine it instead).

There is a blog post from Excentis about how some NAT proxies break half-close.

Nmap's ncat breaks half-close, I have submitted a patch but it has been ignored.

Ngrok breaks half-close, I found out because people tried to solve the Protohackers Smoke Test using ngrok, and couldn't.

If you're writing any kind of proxy, please implement half-close properly.

Conclusion

It's not that hard to read from a TCP socket, but it is easy to get it subtly wrong if you don't have the right mental models.

And (this is turning into more of a Protohackers ad than intended, but) if you want to test your understanding, you could try solving some of the Protohackers problems. If you really want a challenge: Problem 7 has you implement a basic TCP-like protocol on top of UDP.

I first learnt of this idea from a post Why are things? by Billy Bradley (which I recommend reading). He talks about a function U(P,t) which tells you the state of the universe with parameters P at time t. This formalism is useful as an intuition pump, but neither necessary nor sufficient for explaining the core idea. It's not sufficient because relativity tells us there is no universal "t", and it's not necessary because we don't need to define the form of the function in order to talk about what could happen IF a function existed.

I'm also aware that Max Tegmark has done a lot of thinking on this, though I didn't get very far in his book because I got bored of the classical physics long before it got to the interesting part.

The crux of the "mathematical universe hypothesis" is that IF it is in principle possible to define a mathematical structure which completely simulates the universe, then the mere "existence" (in an abstract mathematical sense) of that structure is enough, on it's own, to give the observers within the universe all of their observations. This means that there is no need for the universe to actually be instantiated anywhere (either "physically" or in a computer). And, further, that even if the universe were instantiated anywhere in particular, that this would have no effect whatsoever on the contents of the universe, that we would have no way to find out whether it was being simulated or not, and that even if the simulation were turned off, we would continue to exist.

The universe doesn't stop when the simulation is switched off for the same reason that arithmetic doesn't end when your calculator is switched off. The calculator is not creating arithmetic, it's just exposing arithmetical results to someone who exists outside of arithmetic. Similarly, a universe simulation does not create the universe, it's just exposing the contents of a universe that was always there. For observers within the universe, the existence of the simulation has no effect.

As far as I'm concerned, the argument is utterly watertight: IF the universe can be simulated, then it doesn't have to be. I normally find that people either agree straight away, or won't agree no matter how much we discuss it, but their disagreements don't make any sense and we end up talking past each other.

Please remember the "big if" at the start: IF the universe can be simulated. A counterargument along the lines of "our existence is special and beautiful and can't be created by mathematics alone" is a claim that the universe cannot be simulated.

I do think the question of whether our universe can be simulated is interesting, of course. But I don't know the answer. But IF the universe can be simulated, then that neatly answers a number of difficult questions about the nature of reality:

• who created the universe? (nobody, it exists because it can)
• what is outside the universe? (in the physical sense: nothing; in the abstract mathematical sense: all other possible mathematical structures)
• how did the universe decide when to begin? (it didn't, time as a concept only makes sense from inside the universe)
• how does quantum mechanics work? (the many worlds interpretation)

I think people confuse this argument with a related one: that this "mathematical universe" is an actual explanation of what our actual universe is made of. We know from the first part that IF the universe can be simulated, then that is enough and you don't need any more. So the question of whether our actual universe is made this way is really the question of whether our universe could be made this way. If it could, then by Occam's Razor it is.

I don't know how to find out whether our universe is computable. I expect it probably is, which means I think we do live in a structure that is made out of mathematics.

To argue that the universe could, in principle, be simulated, and therefore could exist as an unsupported abstract mathematical structure, but to then argue that actually that is not what is happening and we really "exist within a simulation on a computer" is akin to arguing that the number 42 lives within your calculator! It is quite obvious that the calculator contains a representation of the number 42, but that is not what the number 42 actually is. 42 existed before the calculator contained the representation, and it will exist after the calculator no longer contains it. The calculator is the simulator, and the numbers are the underlying abstract mathematical structure that reality is made out of.

Consciousness

How can we have consciousness if all there is is mathematics? (I don't know!)

We already have enormously complicated mathematical structures that we evaluate on computers and that we don't understand (ChatGPT, for example). It's not too hard to imagine that a mathematical structure orders of magnitudes more complicated than the observable universe could contain enough hidden complexity to create consciousness. The fact that consciousness exists and that we perceive anything at all is miraculous. But that we don't currently know how to model consciousness mathematically doesn't mean it's impossible.

Free will

If every possible path of every possible universe equally "exists" in the mathematical structure of all things, then we don't really have free will, do we? Probably not. But we also don't "physically exist", and yet we have the experience of physical existence, and we similarly have the experience of free will.

I maintain that everything after "IF the universe can be simulated, ..." is logically sound, which means if you want to cling on to literal free will, then you probably have to make the argument that it is mathematically impossible to simulate the universe.

Computational limits

Physical limits on computation don't apply to the simulation hypothesis as I consider it. Any hypothetical simulation would be outside the simulated universe, so it doesn't have to worry about using more resources than exist within the universe.

But, further, a universe that is "too large to simulate" still exists within the infinite abstract structure of all of mathematics. It exists just as well even if nobody ever figures out a way to evaluate it using their available resources.

Magic

I'm not saying that mathematics contains a representation of the universe, and that some magic process turns that into a real universe that we can experience. I'm saying that the magic process doesn't need to exist. Any "magic", if you want to call it that, is in the structure itself.

The anthropic principle

The abstract structure of all structures is obviously infinite, and contains infinitely many possible universes. Isn't it phenomenally unlikely that we find ourselves not just in a universe in which observers can exist in principle, but that we find ourselves existing as observers ourselves?

Every arrangement of particles that can not observe, can not observe. The fact that we can observe anything at all means we have already selected ourselves into the subset of universes in which observation is possible, and in which we are observers.

Concept

The watch will be about 50mm in diameter, it will only have one hand (the hour hand, obviously), it will be accurate to within 1%, it will probably tick below 10000 beats per hour (very slow for a watch), it will run for almost 12 hours on a single winding, and you'll wind it up and set the time by rotating the outside of the case in the plane of the watch.

Yes, 1% is very poor, almost 15 minutes per day. But I'm not an actual watchmaker, so... lower those expectations.

And, yes, although niche, there is precedent for one-handed watches. And for my purposes it is ideal because the watch isn't going to be accurate enough for minutes to be worth looking at anyway, and I have an ingenious design (if I say so myself) for setting the time that only works if you don't have a minute hand.

I have made a mockup (with no actual works, just ratchets) to show what I mean:

And this is a CAD mockup of a layout for the insides (that I'm not going to use):

The large circle around the outside is the "barrel" and will contain the coiled up mainspring, which provides the power to drive the watch. You wind it up by rotating the outside clockwise, which coils the mainspring tighter. The mainspring pushes against the inside of the barrel, which has the gear teeth around its inside, which then drives the gear train.

The outer barrel will have a ratchet coupling it to the rest of the movement, so that it is allowed to turn clockwise (when you wind it) but will not be driven backwards by the spring tension.

The hand is mounted directly on the inner part of the barrel. This means the only job of the movement is to slow down the rotation of the barrel so that it completes 1 revolution in 12 hours.

To set the time, you need to turn the outer barrel anticlockwise. This will apply torque to the entire movement, through the ratchet that is preventing the outer barrel from unwinding. The movement itself is held into the case of the watch with a second ratchet, which allows it to turn anticlockwise only.

I think this is quite an elegant solution for a one-handed watch. We don't need any actual mechanism to allow the setting of the hand, because we just rotate the entire movement until the hand shows the right time!

The main caveat is that you need the movement to be held in the case with quite a fine ratchet, because you can only set the time as precisely as you can rotate the movement.

The real problem in making a watch is making an escapement. If you can make an adequate escapement, the rest of the watch will make itself. (Or so I tell myself).

One of the biggest accuracy problems in a watch is that the torque from the mainspring reduces as the spring unwinds. This means your escapement needs to be very good, compared to what you could get away with in a weight-driven clock for example.

Fusee

One historical technique is to use a coned pulley called a fusee so that the leverage of the spring is increased as its pulling force is decreased, to cancel out the effect.

(image from Wikipedia)

I don't want to use a fusee. It is bulky and difficult to make, and it becomes unnecessary if the escapement is good enough. The only reason they were used in the olden days is because they didn't have better ideas on how to make escapements.

My simple escapements

My first attempts at making escapements were essentially planar analogues of the verge escapement used in historical clocks and watches. (Incidentally, if you want to see a verge escapement operating, you should visit the clock at Salisbury Cathedral, it is thought to have been made around 1386 and is still running today, albeit with some parts remanufactured in the 1900s).

I wrote an escapement simulator to help me come up with escapements that might plausibly work. Just laying out the geometry of the escapement is a lot harder than it sounds, it is very easy to make designs that jam up, or don't provide any impulse to the balance.

Here's an example of the sort of thing I was hoping to use:

It has 2 pins on the balance wheel, and the escape wheel teeth hit these pins to alternately push the balance wheel back and forth.

This is a very poor escapement because the tick rate changes too much as the drive torque changes. This is because the rate at which the balance wheel accelerates and decelerates purely comes from the escape wheel. An improvement is to add a balance spring, but I spent quite a lot of time experimenting with this sort of thing and I still couldn't work out how to make it accurate enough to work without a fusee.

Lever escapement

So then I moved on to trying to make a simplified version of the lever escapement.

(image from Wikipedia)

The (Swiss) lever escapement is the standard escapement used in almost all mechanical watches for the last 100 years or more, however it has quite a lot of features that need to be formed very precisely in order for it to work correctly.

Pin-pallet escapement

In 1867, someone called Georges Frederic Roskopf invented the pin-pallet escapement as a way to produce lever escapements more cheaply, to use in his "proletarian watch", the first commercially-produced watch for the working classes. You can read more about this watch in History and Design of the Roskopf Watch (mirrored), originally by Eugene Buffat in 1914, translated to English by Richard Watkins in 2007. I first learnt of it from videos by Jacques Favre.

(image from the Watkins PDF)

The pin-pallet escapement is essentially the same as a lever escapement, except that the pallet jewels and impulse jewel are replaced with steel pins. This suits me because I can cut steel pins to length, but I can't (yet!) grind jewels to precise shapes.

However the pin-pallet escapement still has a complicated "guard" arrangement (to the right in the diagram). In normal operation none of the guard surfaces touch each other, so it is tempting to omit them. The issue is that if the watch is shaken, then the lever can unlock from the escape wheel and get "banked" the wrong side of the impulse pin, causing the watch to slam to a halt when the impulse pin swings back a fraction of a second later. The guard pin prevents the lever from moving except when it is lined up with the impulse pin.

Captive lever escapement

My idea was to make the impulse pin captive in the lever. This way even if the watch is shaken and the escapement unlocks prematurely, the lever can never get the wrong side of the pin. My first design with a lever was something like this:

And it worked, to an extent. It was a marked improvement over my previous designs because it allowed an increase in amplitude (e.g. 10° of lever movement are amplified to 40° of balance wheel movement), but it has 2 significant drawbacks compared to a real lever escapement:

1. it does not allow the balance wheel to detach from the lever
2. it does not support amplitudes over 180°

Having the escapement "lock" and allow the balance wheel to swing free from the lever is what allows the balance wheel to keep good time, free from the influence of the drive torque from the escape wheel. And the more amplitude your balance wheel can have, the more time it spends detached, and the less influence the drive torque has on the tick rate.

My design did not support amplitudes over 180° because to do so would cause the lever to pass through the shaft of the balance wheel (yes, we're on shaft passers again). But I persisted with it, because it was better than any of my other ideas.

I added a locking feature to the escape wheel, and this gave another marked improvement in accuracy, enough to convince me that the watch might work, and I set about building a metal version at final watch scale. I've kind of been working on this metal model on and off for almost 2 months now and I still haven't finished it, but this is what I have at the moment:

There is a barrel to wrap a tiny thread around, connected to a gear that will drive the escape wheel, and there is a tiny pallet fork with 0.5mm pallet pins, connected to the balance wheel that has a (very poor) homemade hairspring.

And then one day I had the idea that instead of making all the shafts the same length, I could make the shaft that holds the balance wheel shorter, and let the lever arm pass over the end of the shaft, like this:

Making plates at different heights is a relatively standard idea in watchmaking (a plate that is only supported on one end is called a "cock", so this is a "balance cock"), it just took a mental leap for me to actually perceive that it was something that I could apply. I had been labouring under the misapprehension that for simplicity I needed all of my parts to be pivoted into the same plates.

I made some minor improvements to the pallet and escape wheel geometry, and mainly to the shape of the slot to make the impulse pin more free, and this is my current best escapement model:

It shows amplitudes of well over 180°! Great success. And the tick rate hardly varies even as the amount of drive torque changes by a factor of 4 (from 1 to 4 "nuts" here):

(Note that in the 5 and 6 nut cases, the amplitude was so high that the impulse pin was hitting the balance cock and bouncing back, so the rate increased).

This model was at about 3x the largest scale that I think might fit inside a 50mm watch. Around this time I was experimenting with using a 0.25mm nozzle to 3d print small parts, so I tried to 3d print a version of this escapement at 1x scale.

I have this, which is not quite working yet (the hairspring is from an assortment of clock hairsprings I bought, I didn't wind this one myself):

So the current status on the watch project is that the 3x-scale 3d-printed escapement works well enough to satisfy me, the 1x-scale 3d-printed escapement doesn't quite work yet, but it feels close, and the 1x-scale metal escapement only needs a few more parts (pinion for the escape wheel, something to hold the hairspring to the frame).

Machining

Since I'm not a real watchmaker, I don't have most of the skills that a watchmaker would use to make watch parts, so I have a lot to learn in that regard. But I'm also not constrained to the techniques that a watchmaker would use. As long as I make working parts, it doesn't matter how they get made.

I have a video clip showing how I made my metal escape wheel. The short version is I used a 0.6mm end mill in my CNC router to machine the shape of the escape wheel onto the end of a brass bar, then parted the finished escape wheel off the bar using the lathe.

Clocks

Surprisingly, the watch project has led to a greater interest in clocks rather than watches. I don't know how much of this is just that clocks are so obviously much easier to make.

I enjoyed visiting the Harrison clocks at Greenwich.

I bought a handful of old clock books from a secondhand bookshop in Dawlish, one of them was "How to make an electric clock" by R. Barnard Way, which suggests a design that I think is called a "Hipp toggle". It uses a pendulum to keep time, but the pendulum is given an electromagnetic kick whenever the amplitude gets too low. But I thought the idea of using mains electricity to kick a pendulum is foolish, when you should be able to use the 50 Hz sine wave to count time directly, so I designed and built a stepper motor clock driven directly off 50 Hz from the wall.

I also experimented with what I call the Scotch Yoke escapement. It has no escape wheel as such, instead the pendulum is driven by a Scotch yoke. It will be a poor timekeeper for the same reason that my first watch escapement designs were poor: the pendulum is never decoupled from the drive torque, so more torque will always make it run faster. And in fact it is even worse because the pendulum in the Scotch Yoke escapement always has the same amplitude. But I kind of want to make a clock with it anyway.

Longitude

In case you don't already know the story of John Harrison, the short version is this:

In 1714, the British government offered a prize for anyone who could invent a "practical and useful" way to calculate longitude at sea. It was already known that you could do this with a clock (the difference between solar noon on the ship and solar noon at some reference location tells you your longitude difference from the reference location). However it was not known how to make a clock sufficiently accurate onboard a ship, due to the temperature and humidity variations and the movements of the ship. John Harrison solved the problem, although the Board of Longitude didn't like his solution and he had great difficulty collecting the prize money, eventually receiving most of the prize at the age of 80, having spent almost his entire adult life working on the project.

(The ultimate dream for any creative person has to be to find a project that you love working on the way John Harrison loved working on clocks. I'm still searching.)

If you want the long version of the Longitude story, check out some of the resources I link at the bottom of this post.

Materials

There appears to be some porosity in a couple of the brass plates in H1. It's hard to see clearly because the clock is inside a glass cabinet which prohibits close inspection. It could conceivably be some other sort of damage.

But if it is porosity, then that you would use such poor materials in such an important project suggests that high-quality materials were hard to come by in Harrison's time, or were very expensive. Nowadays we simply take for granted that we can buy sheets of metal that don't have voids in the middle!

Anti-friction wheels

The wheels you're looking at here are not gears, and they don't drive anything. These are anti-friction wheels, a strange type of bearing.

The clocks use anti-friction wheels for shafts that require particularly low friction. You can read in "Concerning Such Mechanism" (mirrored) that Harrison takes the reduction of sliding friction very seriously. Anti-friction wheels are one method of achieving this.

Instead of having the shaft riding directly in plain bearings, the shaft instead sits on a pair of wheels at each end, and rolls against the wheels, and it is these wheels which ride in plain-bearing pivots. The advantage is that you have geared the rotation down by the ratio of the wheel diameter to the shaft diameter (a ratio of 20 or more) by the time it reaches the plain bearings, with a corresponding reduction in sliding friction.

It looks like there is also a lignum vitae bush that will prevent the shaft from falling off the wheels (if it is shaken too violently for example), but in normal operation the wooden bush will not touch the shaft at all.

For parts that oscillate, a complicated version of these anti-friction wheels is used, where instead of building the entire anti-friction wheel, only the part that engages the shaft needs to exist.

Maybe in this recording of H2 you can see the movement in the rectangular aperture as the shaft oscillates back and forth:

It's important to remember that the lower long edge of this rectangular aperture (which is actually a large radius rather than a straight line) is the bearing surface of the shaft. It doesn't just happen to touch the shaft, it is what decides where the shaft sits. It bears the weight of the shaft. At each end of the shaft there are two of these. Obviously the foreground one is easiest to see, but you can kind of see one just behind it, at about 90 degrees to the first.

The part that touches the shaft is on the end of a long arm so that the rotation in the arm's pivot is almost nothing.

Here's a picture of the other end of a similar arrangement on H1:

It took me a little while to work out what the knobs are at the bottom of this picture: they're counterweights for these anti-friction wheel-portions. You can imagine that if there wasn't a counterweight then the weight of the portion of wheel that is present will cause itself to fall down.

A counterweight alone is not quite enough, because while it will make the arm stable, it will be stable in any position, so a harsh jolt could still knock it clear of the shaft. For this reason there is also a spring to return the arm to its centre position.

What an enormous amount of work to go to just to create a pivot for a shaft!

Lantern pinions with rollers

A lantern pinion is a type of gear where the teeth are made out of pins suspended between two plates. Here's an example of a "normal" lantern pinion, from Richard's Stuff (not from Harrison):

Not content with the friction properties of an ordinary lantern pinion, John Harrison put lignum vitae rollers on the pins, so that the friction can be further reduced:

Temperature compensation

The temperature compensation mechanisms of the clocks are the hardest part for me to understand. The picture above is from H1. You can see there is a structure similar to that of a gridiron pendulum (also a Harrison invention) lying horizontally near the middle. A gridiron pendulum is designed so that its endpoints remain the same distance apart regardless of thermal expansion, but in this case I expect it is wired up "backwards" so that it expands much more than normal thermal expansion would account for.

The end of the backwards-gridiron presses against a lever which magnifies the expansion further, which then presses against another lever to magnify it even further, a couple of times, until it eventually adjusts the stops for the balance springs. I think the idea is that to prevent the clock from running at a different rate in different temperatures, the balance spring is forced to stop at a different point.

I have no idea how well it works, but it occupies nearly 50% of the volume of the clock, so Harrison obviously thought it important.

Shaft-passing

There is an interesting feature on H3:

There are stationary pillars that stand in between the spokes of the balance wheels! This is crazy! Why do this?

In the best case it complicates assembly. In the worst case the balance wheel crashes into the pillar.

John Harrison spent 19 years working on this clock, so I guess the answer is just that he did it because he can.

It reminds me of the apocryphal "shaft passer" device, to be specified wherever the spokes of a wheel need to pass through a shaft.

(Incidentally, I also have some thoughts on how a shaft passer should be constructed, and intend to make one one day. See this video with only 2000 views for the best example I've seen so far, although it is not actually a shaft passer per se).

Resources

Some of the best resources for learning about John Harrison, his clocks, and the Longitude problem, include:

• John Harrison and His Timekeepers by Rupert Gould, who restored the clocks in the first half of the 20th Century
• Animations of the Harrison Timekeepers by John Redfern
• Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time by Dava Sobel
• The Clock That Changed the World with Adam Hart-Davis
• "Longitude" the dramatic film adaptation
• "Concerning Such Mechanism" (mirrored): a "translation" by David Heskin, of a work by John Harrison
• "Making a wooden grasshopper clock according to John Harrison's principles" by P. Hefetrueb

I just watched an excellent documentary on mechanical automata. It includes demonstrations (among several other excellent examples) of "The draughtsman" and "The writer" built by Pierre Jaquet-Droz in the late 1700s. These automata are little mechanical boys that can draw pictures and write text, respectively.

I wrote in the previous post on this topic that we don't have any applications, beyond watchmaking, for systems of tiny springs and gears. But these automata are excellent examples to the contrary. Just look at the back of "The writer":

In particular, note the large circle at the bottom: this is how the device is programmed. Each of the shapes loaded around the perimeter of this circle has a different height, and the height changes how far the stack of discs in the middle of the boy is raised up, and the height that this stack is raised up decides the set of discs that are selected, and the set of discs that are selected decides the letter that the writer will draw. The whole stack of discs rotates, each of the boy's "muscles" follows its selected disc, and the boy writes the text that you've programmed in to him. Superb.

This is exactly the kind of stuff I thought we didn't have, and they were making it 250 years ago!

So why don't we make stuff like this today?

Well... imagine how you would go about designing "The writer" if you were designing him today. Unless you're crazy, or have a very specific and unusual set of skills, or want to do it for historical interest instead of practical purposes, you wouldn't do it like this. You'd have a dinky little microcontroller and 3 stepper motors and you'd save yourself a load of time and money and effort and frustration.

The reason people used to make stuff like this isn't that they had tapped into remarkable technology that we're under-utilising now. It's that they didn't have software! If Pierre Jaquet-Droz could have solved his problems using software he would have done so, but he didn't have software so he had to make up for it with incredible ingenuity and hard work.

So the real question isn't "why don't we make better use of tiny gears and springs?" It's "why do we still use gears and springs at all?".

And this also gives us an answer as to why we don't make better use of IC fabs: there's no reason to make anything other than integrated circuits, because integrated circuits are just a substrate for software, and software gets you everything you want.

So why do we still use gears and springs at all? Why don't we do literally everything with software? At some point (for the time being!) we still need to interact with the physical world. For our digital clone of "The writer" boy to actually write on actual paper, he needs to actually move an actual pen, and to do that he'll need motors and probably gearboxes.

But the kinds of springs and gears that are involved there are substantially simplified compared to what Pierre Jaquet-Droz was working with. Instead of making bespoke gears and springs for every application, we make generic gears and springs, and control them with bespoke software, because software is quick and easy to make and mechanical systems aren't.

Obviously, in the long-term, we're all going to be living our lives in permanent VR (to the extent that experiencing the world through the filters of our senses and brain is not already a kind of permanent VR). And then we won't be needing any more motors.

(Also, note that the kind of "pen" that Jaquet-Droz uses is not what we would recognise today, because actual pens hadn't been invented back then. So "The writer" has to automatically dip his quill in the ink periodically while he does his writing! And this is all orchestrated with gears and springs and cams and shafts).

Anyway, the short answer as to why we aren't making maximal use of tiny mechanical systems, or of IC fab technology, is because we can make use of software to solve the same problems faster, cheaper, and easier. It might be interesting to see how that can be extended further: next time you encounter a tiny bespoke mechanical system, maybe imagine how it could be replaced with a generic mechanical system controlled by software.

I have a video demo here:

And you can get the files at https://www.printables.com/model/625901-self-aligning-boggle if you want to print one yourself.

I initially experimented with a system of gears that would rotate the cubes into place, but gave up because I couldn't convince myself that it was going to work. But last week I stumbled across the idea of solving the problem purely using the shape of the cube, with no active mechanism involved.

The only "active" part is that you have to shake it to get the cubes to line up, but you already have to shake it to shuffle the cubes anyway, so I think this is quite an elegant solution.

The cubes are made out of shapes that are like a circle on three corners but a square on the fourth:

With a ramp around the edge of the pockets, to encourage the cubes to fall into place under gravity:

They don't fall into the pockets perfectly every time, but the mesh lid makes it easy to tell if there are any mis-aligned cubes, and you can jiggle it a bit until they fall into place.

I have put steel rods into the centres of the cubes to add weight. This helps a lot with making them fall into place instead of sitting on the edges of the hole, and it also prevents them from bouncing out of their pockets again when jiggling.

I need a better way to paint letters on to parts like these.

Normally I try to make areas that I need to colour stick out, and then it is easy to drag a Sharpie over the high spots, but in this case if the letters stuck out they would interfere with the shape of the cube, so they need to be set in instead. Painting them is very tricky because the paint likes to run in the layer lines, and it is easy to accidentally bump the brush against the edges of the hole. If you have a multi-material machine you could do a much better job.

The mesh lid is very satisfying, it is made out of 20% gyroid infill, with a "modifier cube" in PrusaSlicer to set 0 top layers and 0 bottom layers. It feels surprisingly springy.

I have submitted the design for my self-aligning Boggle board to Spark Hasbro (Hasbro are the makers of the official Boggle game). Spark Hasbro seems to be designed for kids to send crayon drawings of their toy ideas, but maybe they will find self-aligning Boggle interesting.

Mine looks like this:

What is a Douzieme gauge?

"Douzieme" is a French word meaning "one twelfth". As a historical unit, the Douzieme refers to 1/12 of a ligne, which is about 2.256mm. Apparently the "ligne" is still used in French and Swiss watchmaking.

A Douzieme gauge is a traditional watchmaker's measuring tool. Here's a picture I found online:

The legs are squeezed together by hand (against the spring pressure), which opens the jaws. The jaws are placed around the object to be measured, and then the legs are released, allowing the spring to push the legs back open, making the jaws close. When the jaws close around the object, the dimension can be read off the scale. The scale is (say) 5x as far away from the pivot as the jaws are, so the length at the jaw is amplified by a factor of 5 at the scale, which is how the device achieves its precision.

What a simple and ingenious idea! This inspired me to make one for myself.

Making a Douzieme gauge

I made this one:

Pictured here measuring the thickness of a watch gear at 0.55mm.

I don't actually work in douziemes, so I graduated the scale in 0.1mm increments (which are 0.5mm apart in reality, because of the 5:1 ratio).

I opted to make the spring out of stainless steel wire, instead of integrating it into one of the parts, because I made my parts out of mild steel and mild steel springs don't work very well. (For that matter, mild steel jaws probably won't work very well, but I have a solution for that).

I started by designing this in FreeCAD:

I made both of the main parts out of 3mm mild steel sheet:

One of them was made on the CNC milling machine, and one was made by hand on the bandsaw, because I wanted to compare how long each process took. It was 47 minutes for the CNC part and only 26 for the one I did by hand, but I had very suboptimal cutting on the milling machine, and the one I did by hand is less faithful to the intended dimensions and needed more cleaning up with files, so it's swings and roundabouts really.

And I did use the milling machine, even on the handmade part, to cut the step around the pivot. This is the clamping setup I used:

Not great but it was adequate. I should really get a strap clamp set.

I also made a clamping washer on the lathe, and shortened an M4 allen screw to hold the parts together at the pivot.

Then I came to making the spring. My first thought was to make a simple bend in the stainless steel wire and put some holes in the sides of the legs to accept it, but it became obvious that that wasn't going to work (the spring would rotate freely instead of lying in the plane of the tool).

So instead I copied the design from a pair of circlip pliers. I wrapped the stainless steel wire around a screwdriver a few times to form the coil, and hammered the ends over in the vice to make the ends. It works surprisingly well.

I cut out the shape for the scale according to the CAD model, but realised it was not going to work because it is too short:

The pointer is off the end of the scale.

I think probably one of the pieces ended up with the jaw slightly shorter than intended, which means the jaws close closer together, which means the legs end up further apart.

So instead I made a series of paper scales, designed in LibreCAD, to get the perfect alignment:

It is definitely worth going through several paper iterations to make sure the scale is correct.

I found making the scale in LibreCAD to be very laborious, and then I found that LibreCAD doesn't even have any CAM capabilities, despite the homepage stating that "LibreCAD started as a project to build CAM capabilities into the community version of QCad" - it seems it still doesn't have any. It also does not have a constraint solver and is not parametric, it feels like a big step back compared to using the Sketcher workbench in FreeCAD.

If I were going to make another scale like this I would probably write a Perl script to make an SVG file instead.

Once I had a design I was happy with, I made the scale using a diamond drag engraver on the CNC router:

With the engraving filled in with black paint and then sanded back.

And I made a tiny brass clamping piece to hold the scale on with 2x M1.6 countersunk screws:

Why the clamping piece? Why not just countersink the holes in the scale? I made the holes in the scale slightly oversized to allow for positioning adjustment, but if the holes were countersunk this wouldn't work.

The capability for repositioning the scale is my solution for when the jaws wear out: I can file the jaws flat again, and move the scale!

The writing on the scale is very small and some of the digits are distorted (in particular the "8") because of flex in my CNC router. But it is readable.

I am very pleased with my Douzieme gauge, I have built it to a much higher standard than I usually manage.

Line spacing

If you design one of these, then you should take note that the lines on the scale should not strictly be spaced at equal angles, because what you are really measuring is the chord length rather than the arc length. However the error if you space them at equal angles is so small that you can probably ignore it. I did go to the trouble of spacing them correctly the first time, but then realised I had messed up the alignment and couldn't be bothered doing the several-hundred clicks in LibreCAD to rescue it, so I started again and went with equal angles. Maybe if you are better at LibreCAD than me you could do it properly.

Alternative designs

Vernier Douzieme gauge

Instead of just having a pointer on the "moving leg", we could put a Vernier scale on it. That way we'd read the scale with the same precision as a normal Vernier caliper, except the scale is amplified by 5x compared to the object we're measuring, so we'd get 5x the measuring precision of a normal Vernier caliper.

Micrometer Douzieme gauge

Another idea is that instead of the legs being spring-loaded and moved by hand, they could be moved by a screw, and we can read off the length at the legs with a micrometer scale, which again gives us 5x the precision of the micrometer because of the leverage.

Primitive circuit boards

This is obviously an exhibit showing some primitive circuit boards at various stages of manufacture, but I think the exhibit is being updated because there was no obvious explanation posted.

It looks like there is a wooden pattern (out of shot to the left), which is then cast as a metal part, which then gets machined and painted, and then turns into a working circuit board. Eh? Won't everything short out through the middle?

It turns out that's not how it works at all. There is an explanation on the Science Museum website, it's a Passive component plate, type A3, for Sargrove sprayed-circuit radio receiver.

The "wooden pattern" is actually Bakelite, and is not a pattern at all but is the insulating substrate for the circuit. This is then sprayed (how?) with metal and graphite, and then the top is machined down to remove all of the metal except in the grooves!

It's kind of the opposite of a modern circuit board, where the conductive part is raised up off the substrate. In these Sargrove circuits, the conductive part is sunk down.

Apparently "his idea was never taken up generally, partly because it was seen as a threat to jobs", which sounds ridiculous now given how many modern jobs are dependent on our ability to produce cheap circuit boards.

James Watt steam engine indicator

This is a device for plotting pressure-volume diagrams from steam engines. I have done this for my Wig-Wag engine by attaching electronic sensors, but this device works purely mechanically.

The central body is a spring-loaded pneumatic cylinder, which gets plumbed into the steam engine's cylinder pressure. There is a pencil mounted horizontally at the top of the "plunger", so as the pressure in the engine rises, the pencil rises.

The piece of card is attached to a piece of string, which is biased to the right by the weight, and the hook on the left end of the piece of string is presumably attached to something on the engine which moves in proportion to the stroke.

Now as the engine completes a cycle, the card moves left and right, and the pencil moves up and down, and plots a representation of how the pressure in the engine changes throughout the cycle. Ingenious.

As a bonus, just at the edge of the shot you can see the Watt micrometer. There is an interesting Machine Thinking video on this device.

Electric governor

It was quite hard to take a photograph of this, but maybe you can see what I'm getting at.

This is a steam-turbine-powered electricity generator. The output voltage is applied across a solenoid which pulls a metal rod (right of pic, vertical), and the rod is connected to a long lever which actuates a steam "throttle" (left of pic), to create a feedback loop which keeps the output voltage constant.

This is a nice alternative to the centrifugal governor, and it directly controls the output voltage, which is what you care about, rather than the engine speed.

Rotating oiler

Two oil cups are mounted on the railing around the outside of the engine, with a brass pipe poking into an "eye". The eye is on the end of a long hollow arm attached to the crank pin. Oil that drips into the eye can run down the inside of the arm, into some (presumed) oil ways in the crank pin, and lubricate the crank bearing. The arm keeps the eye on the centre line of the crankshaft even though the crank pin moves around in a circle.

It looks like this is actually not used, because both of the oil cups on the railing are empty, and there is a third oil cup installed directly on the con rod, but it's a nice idea anyway. I guess it is prone to leaking because the arm spends 50% of its time upside down.

Gnome rotary engine

There is an area in the "Flight" section on the top floor full of aircraft engines. This area is sadly very poorly lit which makes it hard to see much more than "metal parts are here" unless you look closely. I think it's more meant to provide an aesthetic than be an actual exhibit.

But the Gnome engine is worth an exhibit in its own right. The example at the Science Museum is cut away so you can see what's inside.

I didn't think to take any pictures, but here are some I found online:

Instead of mounting the crank cases to the fuselage, and the propellor to the crankshaft, this engine works the other way around. The crankshaft is fixed to the fuselage, and the propellor is attached to the crank cases. When the engine is running the entire thing spins around with the propellor! Supposedly this helps with cooling.

But it's not quite as simple as that. If you turned around an ordinary engine so that the fuselage and the propellor were swapped, you would have issues connecting the throttle cable and fuel. The Gnome engine supplies fuel/air mixture through the inside of the crankshaft. The exhaust valves are on the top of the cylinders and are operated with push rods. The inlet valve is in the top of the piston and is spring-loaded with no actuator. The valve simply opens when the pressure in the cylinder is lower than the pressure in the crank case, and shuts when it's the other way around. Ingenious!

The Gnome engine is French and is known as a "monosoupape" (single valve) because there is only one obvious valve per cylinder.

I have a vague idea for a model steam engine that works a bit like this. Each cylinder would have a single port through which the steam both enters and exits, like in an oscillator. Except instead of oscillating back and forth between an inlet port and an exhaust port, the cylinders would be continuously rotating, fixed to the flywheel (perhaps even bored directly into the flywheel), and the port would be exposed to steam pressure for ~half the cycle and vented to atmosphere for the other half.

I think one difficulty would be in sealing the port face. Oscillating engines can get away with just using a spring because the rotation is minimal. The "rotary" version could maybe use a spring but would need a thrust bearing to allow the spring to rotate.

I want to make a Bangle.js commute timer, as kind of a complement to Waypointer Moto: instead of telling me where to go, it will tell me how long it's going to take. It will help me answer questions like "how many minutes did that horsebox really cost me?", and "is the traffic really worse today or am I just more grumpy?".

I don't just want a rough minute-precision ETA like you get on Google Maps. When I'm stuck at traffic lights I want to see my delta time ticking up by 1 second per second. I want to really feel that delay.

This will be very much a home-cooked app. It only needs to solve my problem, it doesn't need to be useful for anyone else. I might even hardcode the route endpoints.

It's easy to say "make it store a reference route and show your delta time and ETA, and update the reference route if you beat the time", but there are a few subtleties to the implementation that make it harder than it sounds. The purpose of this post is to come up with a design that addresses all of the subtleties, so that I can write it right on the first try.

Problems

Sometimes the Bangle.js won't have picked up a GPS fix until after I have set off.

Sometimes it will have a GPS fix before I set off.

Sometimes roads are closed, or I need fuel, or the GPS stops working temporarily, or I deviate from the route for some other reason.

Sometimes I will loop back on myself, or the GPS signal will be noisy.

Sometimes I'm going home instead of to work.

Sometimes I will be standing still or moving very slowly.

When the new route is faster than the reference route and we update the reference route, the start and end points should stay fixed even if the GPS trace has a small amount of drift.

Sometimes the time on the Bangle.js drifts, particularly when the battery is low.

Solutions

The GPS route will be a series of waypoints annotated with timestamps. This could be stored in GPX format, but wouldn't have to be.

If the commute is about an hour long and we want under 1000 waypoints, then we want to take waypoints less frequently than once every 3.6 seconds. Maybe a 5 second interval would be fine. Why 1000? Just an arbitrary starting point. Note that the Bangle.js does not have an awful lot of RAM and ideally we'd keep the entire route in memory.

Whenever we get a fresh location, we'll look for a "target" waypoint, which we define to be the nearest waypoint which we are getting closer to. (Finding this could be O(1) in the common case, where you have moved a bit closer to the same waypoint you were closest to last time, but O(n) in the worst case, if you have to scan all waypoints).

If the target waypoint is the first waypoint of the route and we're more than, say, 20 metres away, then we'll say that we haven't started the route yet.

Otherwise, once we have the target waypoint, we can look at our current speed and our distance from the target to work out how long it will take us to get there. We then know a point on the reference route, and the time that we expect to get there, so we can work out how long it would take us to complete the route at the reference pace, and therefore what time we should arrive at the destination. (And if we have so far averaged 1% slower than reference pace then we can also assume that we'll continue to average 1% slower and adjust the ETA accordingly.)

I think the only time this target waypoint selection falls down is if we're moving very slowly, and the noise in the GPS data could cause it to think we've moved away from the target, which means it would select a new target in the opposite direction. Maybe we say that if we're moving less than, say, 5mph, then the target selection works differently (e.g. bias it towards retaining the existing target even if it looks like we moved away from it).

There's also the issue that if we're not moving, then the time to the target waypoint is infinite, so we expect never to arrive at the destination. This is obviously wrong, so maybe we cap the "time to next waypoint" at the total time it took to reach that waypoint in the reference route.

So this waypoint selection method allows us to "wake up" partway through the route and still select an appropriate target. It allows us to take a diversion for a road closure and rejoin the route with an appropriate target as we approach it. If we make sure that the "start" location for the route is 100 metres or so up the road from my house, then it also won't accidentally start the timer while I'm still faffing about on the drive.

But how do we calculate the starting time for a journey where we woke up a few miles into the route? I propose that if we "start" the route with some target other than the starting point, then we blindly import all of the preceding waypoints into the "current" route, but adjusted to match today's timestamps. That way we end up with a complete route that we can use as the new reference if required, and we have an estimate of the full journey time. Another option would be to sit on the drive until the watch has a GPS fix, but that is less appealing.

To prevent the start/end point from drifting (e.g. if we start the journey when we're within 10 metres of the start point, it would move back by up to 10 metres every time a new reference is taken), we can maintain the original coordinates for the first and last waypoint, and adjust the time to match our best guess at when we passed that exact point on the new journey.

When the GPS data is noisy we might find that the delta time and ETA become noisy. We could update them using an exponential moving average instead of replacing them each time there is new data. Will have to experiment with that.

And to solve the issue where we're going in the reverse direction, we can keep separate reference routes for each direction. It might be good enough to decide which one to use based on time of day (i.e. before noon we're going to work and after noon we're going home). Alternatively we could keep track of the delta time for each route and say that we're travelling the route whose delta time looks most stable. I think just looking at time of day is adequate and simpler.

To keep the Bangle.js time in sync I will make it update the time from the GPS fix whenever there is one.

Display

The key pieces of information I want to display are:

1. GPS status (high-quality fix, low-quality fix, or no fix at all)
2. ETA as a time of day, maybe with sub-second precision (e.g."08:57:43.1")
3. delta time to personal best, definitely with sub-second precision (e.g. "-1.5")

After that, there is lots of other stuff that we could easily calculate: elapsed time so far, estimated time remaining, estimated total time, average speed, average speed delta, start time, current time of day, progress percentage in terms of time, progress percentage in terms of distance, probably lots more.

I might experiment with showing some of that, but the ETA and the delta time are the most interesting to me.

Once we arrive at the destination, we can switch to a different mode that shows the overall delta time and the true time of arrival. If the overall delta time is negative (i.e. we beat the best time), then it should probably update the reference route to use the new journey.

But maybe we would want that to be optional, for example if the journey was fast for some non-repeatable reason that we don't want to confound the data on future journeys? And for that matter, maybe we'd want the option to replace the reference route even if the new journey was not an improvement? Not sure.

Conclusion

I'm not sure whether writing all this up has made things clearer in my mind or less clear. And I still don't know what to do about the case where the reference route has loops in it. Maybe it's fine to ignore as an unlikely edge case? Maybe before storing a reference route we need to post-process it to cut out loops? Maybe it's as simple as deleting any waypoint whose successor under the "target selection" logic has an earlier timestamp?

It has electronics hidden inside the wood which can detect the coins and then transmit the readings over Bluetooth, which are then picked up by my Bangle.js smartwatch, which vibrates to indicate the location of the coin.

It uses what I believe to be 2 novel innovations in magic:

• a method for detecting coins through solid wood that can be packaged extremely thinly
• a method for making sheets of plywood with electronics hidden inside

Presentation

The box is fully examinable by the audience. For them to prove that it is not made out of ordinary plywood they would need to either take the hinges off, or somehow X-Ray it. It is otherwise indistinguishable from ordinary plywood. I thought that this would make it an extremely powerful trick.

However it might be more of a "magician fooler" than a genuinely good trick, because a lay audience just needs to think "the box is rigged" to think they've figured out how it works. Sadly we'll never know how good it actually is as a trick, because I don't know any actual magicians who might be able to perform it. I have performed it myself for friends a handful of times but it did not get the kind of reaction I was hoping for. I don't know how much of that is because it is obvious that the box is rigged, and how much is due to my poor presentation.

I have been reading The Jerx magic blog lately. It has a lot of really good information about performing magic in a social setting. The key idea is that instead of doing overt magic "performances", you want to be secretly setting up magical moments that you act like you weren't responsible for. That way you avoid creating a division between the audience and the performer, and it fits more naturally in ordinary social situations. It works much better than having everyone pause the party to watch you stumble through your unnecessarily-long lecture on Egyptian faith healing. Or so I imagine.

Detecting coins

The most interesting part of the trick is how it detects the coins. There is a copper plate underneath each coin location, and another copper plate above. These plates form a capacitor, but the gap is quite large and the capacitance is extremely low. When a coin is placed in between the plates, it acts as if the plates have been brought closer together, and the capacitance increases. By measuring how long it takes to charge up the capacitor, you can tell how much capacitance there is, and therefore whether or not there is a coin in between the plates.

I prototyped this with a pair of 3d-printed parts. Each part holds a piece of copperclad board, and one part has a cutout for the coin.

It worked well enough to convince me that I could proceed.

Making plywood with electronics hidden inside

You can make plywood with electronics hidden inside by starting with 2 pieces of plywood that are half the thickness you require, milling out a pocket for your electronics, gluing the electronics inside, and then gluing the two pieces together.

You'll want to make sure that the mating faces have perpendicular grain directions. And once the two pieces have been glued together, you'll then want to cut your piece of magic-plywood down to the actual shape you require so that the join is seamless.

Now you have the problem that your battery is glued inside the wood and you can't access it to charge it. I left a small gap in the wood through which I can poke a USB-C charging cable, hidden behind one of the hinges.

Gluing a lithium battery inside a piece of flammable material is probably a safety hazard, so I put a 1A "polyfuse" on mine. This is a type of fuse that automatically resets after some time has passed, very handy since I wouldn't have any way to replace a blown fuse. 1 Amp is way more than I actually need, a lower current rating would probably be better.

Implementation details

Capacitor plates

Putting the copper plate underneath the coin is easy, I made a circuit board with the CNC router:

But how do we get a copper plate above the coin? I put a solid sheet of copper connected to a common ground plane inside the lid of the box, and conducted it to the ground plane of the main PCB through the hinges of the box! The solid sheet in the top has a wire soldered to it which is soldered to one of the hinges:

And the PCB on the bottom has a pogo pin to connect its ground plane to the hinge:

(Why not solder that one as well? I wanted the hinge to be removable so that I can access a pin header to disconnect the battery).

Circuit board

The microcontroller I used is a Seeed Studio Xiao ESP32C3. It is a good choice because it is very small, and it has a built-in Bluetooth radio, and a built-in lithium battery charging controller. That meant I hardly had to put any components on the board:

We have the microcontroller board, rectangular sticky-back Bluetooth antenna, 5x 1-Megaohm resistors for the capacitor charging circuits, a grey bodge wire (see below), a red wire connecting the battery charging circuit to the positive terminal of the battery, a pogo pin, a pin header to disconnect the battery, the battery, and a polyfuse.

Measuring capacitance

The Arduino code I use for measuring how long it takes to charge up a capacitor is:

int sensor(int n, int m, int sendpin, int recvpin) {
  int counter = 0;
  for (int i = 0; i < n; i++) {
    digitalWrite(sendpin, HIGH);
    for (int j = 0; j < m; j++)
      if (!digitalRead(recvpin)) counter++;
    digitalWrite(sendpin, LOW);
    for (int j = 0; j < m; j++)
      if (digitalRead(recvpin)) counter++;
  }
  return counter;
}

n sets the number of times we charge it up/down, to help with averaging the result. m sets the number of loop iterations we wait for it to charge up/down, sendpin is the pin that is connected to the "positive" capacitor plate through a 1-Megaohm resistor, and recvpin is a pin connected directly to the positive plate (the negative plate is tied to ground).

As the charge current slowly trickles out of the sendpin and through the Megaohm resistor, eventually the plate will get charged up enough that the recvpin goes high. In principle we could stop the loop as soon as the pin goes high, but in practice I found the readings to be quite noisy. Running a fixed number of iterations and counting how many the pin is low for works better. I also found it useful to measure how long it takes for the capacitor to discharge. You need to wait for it to discharge regardless, you may as well get some data from it.

If you use a Seeed Studio Xiao ESP32C3: don't try to use pin D9, it is internally connected to ground through a 300 kOhm resistor, which means you can never charge the capacitor up enough for it to work. I fixed this by disconnecting that pin from D9 and ran the grey wire to connect it to D0 instead.

Bluetooth

For some reason I was unable to make the ESP32 and the Bangle.js reliably connect to each other over Bluetooth. There was some speculation on the Bangle.js forum that this was caused by one of the devices being slightly out of spec on the Bluetooth timings. I don't know what the problem is. But since I only needed one-way communication, I had an ingenious solution.

Instead of connecting the watch to the ESP32 to receive sensor readings, I made the ESP32 write the sensor readings into the "manufacturer data" field of its Bluetooth advertisements! The watch can then read the values out of this field without ever having to actually "connect" to the ESP32.

I wanted to give it a better name but everything I could think of was already taken by too many other people's weird hacky DNS tools.

Why?

Yesterday I was trying to test how a piece of software would behave if it received multiple A records for a given domain name, but not always in the same order. I couldn't change the DNS records for the domain in question, and setting up an alternative to test with is a lot of faff. Wouldn't it be easier to just type something like:

$ sudo dnstweak foo.example.com=127.0.0.1,1.2.3.4

And have it do what I want?

Well I didn't know of a tool to do that so I made one myself.

How?

It's written in go and has only one (immediate) dependency: Miek Gieben's dns library. The dns library made it very easy to write this program.

At startup it grabs the first nameserver line from /etc/resolv.conf (to use as the upstream resolver), and all search lines, stores the existing resolv.conf in memory, and writes a new resolv.conf that persists the search lines, and points nameserver at itself. When it exits, it restores the resolv.conf that it has stored in memory.

When a spoofed response has more than one IP address, they are shuffled. (Just because this is the behaviour I wanted to test - but it would be easy to make this configurable).

One cool feature is that it looks up the client addresses under /proc to work out which process each request originates from. I didn't actually need this feature but it was fun, and easy enough to do. You need to look in /proc/net/udp to find the inode corresponding to the client's address, and then look through all of the /proc/$pid/fd/* to find one that has the same inode, then you have found the PID of the client, and you can look in /proc/$pid/cmdline to find out the name of the process.

Why not?

The most fragile part is that if it crashes then it won't restore your original resolv.conf. I recommend taking a backup of your resolv.conf before you start. Probably dnstweak should make such a backup itself, but would need to be careful to make sure a second run of dnstweak doesn't overwrite the first (good) backup with a copy of the resolv.conf that the previous run created!

Also, maybe it should detect when systemd is managing the nameserver and try to cooperate with systemd instead of unilaterally trampling over resolv.conf. And maybe it should watch /etc/resolv.conf with inotify so that it can at least warn you if something else changes it back.

Example session

Open 2 terminal windows. We'll run dnstweak in the first and ping in the second.

First terminal:

$ sudo ./dnstweak example.com=127.0.0.1
2023/07/11 20:41:35 dnstweak starts
2023/07/11 20:41:35 listening on 127.0.0.1:53
2023/07/11 20:41:35 using 127.0.0.53:53 as upstream resolver

dnstweak stays running. It has inserted itself as the local resolver by modifying /etc/resolv.conf.

If you run dnstweak as a non-root user, it will still work as a DNS server, but it won't be able to listen on port 53 or insert itself into /etc/resolv.conf.

In the other terminal, start a ping to example.com:

$ ping example.com

Now we get some log output from dnstweak in our first terminal:

2023/07/11 20:41:37 127.0.0.1:48439 (ping/9975): A example.com: 127.0.0.1 (overridden)

A request came from 127.0.0.1:48439, which is a process called ping with PID 9975. It was an A lookup for example.com, and we returned 127.0.0.1. Back in the other terminal:

PING example.com (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.020 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.037 ms

Great success.

More

There is more information, and Linux x86-64 binaries, in the github repo.

Context

OpenSIPS is an open source SIP proxy. The topology_hiding module hides information about your network topology.

From the Topology Hiding tutorial:

Topology hiding is usually utilized as an approach to enhance SIP network security. Since, in regular SIP traffic, critical IP address data is forwarded to other networks, the concern is that third parties can use that information in order to direct attacks at your internal SIP network.

If topology hiding is used as a security measure, then you might naively think it is a security vulnerability if it is not hiding anything.

However, I reported my findings to the opensips-security mailing list, and was informed that this is not a security issue. I think I just don't know enough about SIP, but that's fine. It's still fun to exploit it.

The encoding

If you're using topology_hiding without the dialog module, then the topology information is encoded into a URI parameter of the outgoing Contact header. (If you are using the dialog module, then my understanding is that OpenSIPS stores the information internally and does not pass it on even in an encoded form).

The 3 things that are encoded are the incoming packet's Record-route set, Contact header, and socket address. This is done by the build_encoded_contact_suffix() function.

The information is serialised into a string, the string is encrypted, and the result is encoded with a slight variation of base64, before being passed out as a URI parameter in the outgoing Contact header.

Here's an example serialisation (my example has no Record-route set)

\x00 \x00                            Record-route length (0)
\x23 \x00                            Contact length (35)
sip:foo@10.0.0.5:9000;transport=udp  Contact
\x12 \x00                            Socket length (18)
udp:127.0.0.1:8000                   Socket

The lengths are big-endian.

And here is the code that encrypts the string:

for (i=0;i<(int)(p-suffix_plain);i++)
    suffix_plain[i] ^= topo_hiding_ct_encode_pw.s[i%topo_hiding_ct_encode_pw.len];

(suffix_plain is the string we're encrypting, p points just past the end, and topo_hiding_ct_encode_pw is the password from the config file).

It XORs each byte by the corresponding byte of the password, looping the password when it is too short. This is the Vigenere cipher, but with XOR instead of addition.

Example

Here's an example of how you might configure the topology_hiding module in your OpenSIPS config:

loadmodule "topology_hiding.so";
modparam("topology_hiding", "th_contact_encode_param", "thinfo");
modparam("topology_hiding", "th_contact_encode_passwd", "m6d90nmvg645fh");

This will put the encoded data in a Contact URI parameter called "thinfo", encrypted with the password "m6d90nmvg645fh".

Here is an example of a Contact header that OpenSIPS could transmit:

Contact: <sip:jes_test_account@10.0.0.9:6000;thinfo=bTZUOUMHHUwNU0dqEg0eQjtYUw0CAwlCdARWRl0YVBcJVFtGVwYPQRQJA0UUVkIaUAIERiY1EwwdDFULB0BdWFcYBQ9TWF0G>

Cryptanalysis

Given that the incoming and outgoing Contact headers have the same user identifier (probably not true in every case, but common), and given that the incoming Contact header is part of the plaintext that was encrypted, we have quite a large fragment of known plaintext.

If we assume the absence of a Record-route set, then we know that this fragment starts at the 3rd byte, and we also know that the first 2 bytes are both zero (i.e. the ciphertext equals the key for those bytes). So it is possible to simply XOR the start of the ciphertext by the fragment that we know, and we retrieve a repeating copy of the key, which we can then use to decode the rest of the ciphertext.

Even if there is a Record-route set, all is not lost! We know that our known plaintext fragment is in there somewhere, so we just need to try all possible key lengths and locations of the fragment until we find one that does not present a contradiction. This runs in milliseconds.

I have written a proof of concept.

$ time ./th-decode.pl
key="m6d90nmvg645fh" msg="\x00\x000\x00sip:jes_test_account@10.0.0.9:6000;transport=tcp\x12\x00udp:127.0.0.1:5000"
 
real	0m0.015s
user	0m0.011s
sys	0m0.004s

Even if we didn't have a long known plaintext fragment, there is plenty of scope for an attack. You at least know the form of the plaintext (e.g. you know the lengths have to add up, you know what a socket address looks like, you know that a Contact header begins "sip:" and has an "@" sign in it).

Even if the key is longer than the message, we just move into the realm of key reuse with a one-time pad, which is still no bueno. If you can collect two ciphertexts and XOR them together then you are left with the XOR of the two corresponding plaintexts, at which point you don't care about the key at all.

Mitigation

Since I learnt from the opensips-security mailing list that this is not a security issue, no mitigation is necessary.

That said, if you were inclined to mitigate the problem, the best solution is probably to switch to using the dialog module so that your topology information is kept inside OpenSIPS.

Also, while I have your attention, OpenSIPS does not verify hostnames in SSL certificates. I wrote a fix but it's a bit clunky, and nobody who knows better seems to care to fix it, so it's probably not going to get fixed any time soon.

But then there are some technologies that just seem so incredibly far advanced compared to other things that are possible, but also are only applied to incredibly specific applications.

Watchmaking is one example. We can make these unbelievably tiny systems of springs and gears that can keep time quite accurately, and then when you look for other things using the same sort of technology, there's just... nothing. Really? There isn't a single application of unbelievably tiny systems of springs and gears apart from making watches?

Integrated circuits are another example. For some reason we can make ICs with such precision that we're almost placing individual atoms, but it can only be used to make integrated circuits? There's literally nothing else we can make with this technology?

Feynman wrote about "micromachines", where IIRC the idea is you make a machine that can produce slightly-smaller versions of all of its own parts, and then repeat the process until the machine is arbitrarily-small. Obviously it's harder than that makes it sound, because the tolerances have to go down in tandem with the overall dimensions, but I don't see why in principle it wouldn't work.

I think part of the problem is that watchmaking and IC fabrication are both highly specialised fields, and people outside these fields kind of forget, or don't know, that the techniques are possible. If someone outside watchmaking wants to design a small mechanism, and they start imagining gears and springs in their head, the smallest reasonable-seeming scale is still maybe 10x larger than what is typical in watchmaking. You just wouldn't assume that you can pack dozens of gears into an object about the size of a large coin. You wouldn't even consider such a design, it is so obviously outrageous.

Now I don't personally have any designs in mind that can make use of watchmaking-scale gears, or IC fab-scale whatever-sort-of-stuff-it-is-that-ICs-are-made-of, but that's exactly the point: there aren't any other applications because people don't have any other ideas, but I think the reason we don't have any ideas is because we forget that it's possible. So I think it would be worth trying to bear in mind that these things are possible, so that we don't overlook possible inventions that could pop into our heads.

If you want to try encoding some strings, you can try my Triangle Strip Encoding tool. I did not make a decoding tool, so you'll have to do that on your own.

Here's the ring I made:

And the pattern on it:

How it works

First, turn your data into a bitstream any way you like. I simply concatenated ASCII codes.

0100101001010011

Now prepare a grid with uniformly-spaced vertical and horizontal lines. You need as many vertical lines as you have bits to encode, and you need 4 horizontal lines.

Now plot your data on the grid, using the top 2 lines for even-indexed bits and the bottom 2 lines for odd-indexed bits, and draw lines connecting all the points:

Now draw lines to connect all the even-indexed points:

And finally draw lines to connect the odd-indexed points:

Lose the grid and you have your abstract-looking triangle pattern:

Ambiguity

My ring encodes 0100101001010011 which is ASCII for "JS".

But a ring has no natural "up" orientation, so there are actually 2 possible ways to decode it. One way gets you the "JS", and the other gets 0x35 0xad, which is a digit "5" followed by a non-ASCII-code.

In general, if you get it the wrong way up, then your bitstream is reversed and inverted. It might be fun to find English words that give other valid English words when you reverse and invert their bits.

Further expansion

You can encode your data in some base other than binary, and use more horizontal lines. For base-n, you need 2n horizontal lines. (It would actually be possible to use a different base to encode the even numbers vs the odd numbers, see my post on chess steganography for an example).

Using a larger base might give you more interesting-looking triangles, and it would definitely give you shorter sequences, at the cost of making the decoding harder to eyeball in the absence of the grid.

Also, you don't need to put the values in any particular order, and the order of the values for the top line and bottom line do not need to match. For example:

Finally, you could run several different strips horizontally, and then connect the triangles between them, to make a triangle mesh that encodes your data:

The notices were actually sent to abuse addresses at DigitalOcean and gandi, and I think gandi forwarded them to me.

I have now taken hardbin.com down completely because dealing with this sort of thing makes it less fun to run and more like hard work, but I do still have a copy of the log files.

I did some bash-fu to extract the IPFS hashes from the emails and grep for them in my nginx logs, and was surprised to find not a single match.

In case you are interested, I have posted the contents of the takedown notices in github gists:

• https://gist.github.com/jes/51496baaa48610f1b59a39804fd28df9
• https://gist.github.com/jes/597cf1fa84067586c906ac1d8c605f20
• https://gist.github.com/jes/f32147236874ef736be6190c2cce4a3d

Graham pointed me at a Law.StackExchange thread covering what happens if you send false DMCA takedown notices. The short answer is I think nobody has ever faced criminal charges, and there have been a very tiny number of civil cases.

The emails are sent from "notice@ciu-online.net". There is a login form on www.ciu-online.net, but it is incredibly generic, so it's hard to work out who is behind it:

The name at the bottom of the emails is "Gareth Young, Internet Investigator". I did find a slideshow on Creating a Global Internet Anti-Piracy Strategy by a "Gareth Young - Senior Internet Investigator" who apparently worked for Covington & Burling LLP, although the name Gareth Young does not currently appear in their A-Z list of "Professionals". It is certainly possible that this is nothing to do with Covington & Burling LLP, I have nothing but that slideshow to suggest any link to that company.

Page 14 of the slideshow, "What Are Your Options?" includes "Make it less fun to run and more like hard work".

Certainly the receipt of these DMCA takedown notices has made hardbin.com less fun to run and more like hard work, albeit that nobody was using hardbin.com to pirate any of the books he listed (so it has had no effect on piracy whatsoever), nor indeed did he even check whether hardbin.com could be used to access the URLs he listed, because there were no requests in my logs for any of the 7350 URLs. I checked a few, and they didn't work, they just gave 504 Gateway Timeout responses.

Apparently the DMCA requires "a statement that the information in the notification is accurate". So what does the information in the notification actually say?

3. Statement of authority:

I swear, under penalty of perjury, that the information in the notification is accurate [...]

(The only part that the DMCA states is under penalty of perjury is the assertion that the complaining party is authorised to act on behalf of the copyright owner. As far as I can tell, there is not in fact a penalty of perjury for putting other false information in a DMCA takedown notice. But never mind that part.)

Gareth Young, Internet Investigator, swears, under penalty of perjury, that the information in the notification is accurate. But section 2 begins:

2. Copyright infringing material or activity found at the following location(s):

Copyright infringing material or activity could not have been found at those locations because in order to find it you must have accessed it, and since my logs show that nobody accessed it, we can infer that Gareth Young, Internet Investigator, can not have found it. It is not enough to make the leap from "this content exists at this IPFS hash" to "this IPFS gateway allows access to this content" because you don't know if the hash is already blocked in the web server configuration, or if the server is broken in some way that prevents access to that hash, etc.

I believe this amounts to inaccurate information in the DMCA notice.

I found an ipfs-gateway-dmca-requests github repo which documents a similar phenomenon on a different IPFS gateway, although it seems they've been subject to it since at least the 15th of February 2022 - over a year ago.

The README in that repo also suggests that that person's DMCA requests also usually come from someone called Gareth Young, and are also sent without first checking whether the allegedly infringing URLs are actually infringing.

So are these DMCA takedowns fraudulent? And, if so, what can we do about it?

Emma pointed out that an impossible crime is what I wanted and an impossible crime is what I got!

The puzzle

The Bloodhounds are a group that meets weekly to discuss crime fiction. Milo Motion, a member of the group who lives on a boat, is due to read from a particular chapter of a particular book at the next meeting. In the intervening week, a valuable stamp is stolen from the postal museum. At the next meeting, when Milo opens the particular book, the stamp is discovered at the particular chapter, as if it were a bookmark. Milo goes straight to the police station to hand in the stamp, and he didn't have time to go anywhere else in between. The other members (including a man called Sid Towers) leave the meeting early. After being interviewed for a few hours, Milo is escorted to his boat, under constant supervision from police, unlocks the boat, and finds the dead body of Sid inside!

The padlock on the boat is said to be very high quality (German-made, in fact), and shows no signs of damage. There were only ever 2 keys for it, Milo had constant possession of one, and the other was dropped in the river a long time ago. And lock-picking does not exist. When Milo is in the boat, he leaves the padlock hanging outside and locks the door with a bolt from the inside, so nobody could have snuck in while he was asleep.

The solution

We learn that a disgruntled former member of the Bloodhounds, Gilbert Jones, still bears a grudge after being snubbed by the group. He discovers that his girlfriend of 6 months, Shirley-Ann Miller, enjoys crime fiction, and subtly leads her into joining the group. He learns from her that Milo is going to read the particular chapter of the particular book at the next meeting, and decides to steal the stamp and plant it in Milo's book to prove (to whom?) that he is smarter than the Bloodhounds.

Fine so far. The stamp theft is relatively straightforward: he climbs up a ladder posing as a window-cleaner, breaks in through a window, and takes the stamp before the museum opens. But how does he get the stamp into Milo's book?

Gilbert buys a lock that looks identical to Milo's padlock. At some time while Milo is inside the boat, with his padlock hanging outside (unlocked!), Gilbert simply switches the locks. When Milo leaves the boat, the lock he clicks closed is Gilbert's lock. Gilbert hides and waits for Milo to leave, then unlocks his own lock, using his own key, and gains entry to the boat. He locates the particular book and places the stamp in the particular chapter, leaves the boat, and locks it with Milo's lock, leaving no sign of any tampering. Milo eventually returns to the boat, unlocks his own lock, and doesn't find anything wrong. Eventually he takes the book to the Bloodhounds meeting, whereupon the stamp is discovered.

Superb! Top marks so far.

When Milo is at the Bloodhounds meeting, Gilbert is back at the boat to put Milo's lock back, but he doesn't know that the discovery of the stamp will lead to the meeting ending early. Sid's curiosity compels him to investigate the boat while Milo is at the police station. Sid finds the boat unlocked, and Gilbert is inside. Gilbert panics and hits Sid over the head with a windlass, accidentally killing him. Gilbert locks the boat with Milo's lock and runs away.

What?!

The flaws

Gilbert can't have been at the boat to return the lock while Milo was at the Bloodhounds meeting. In order for Milo to have collected the book with the stamp, Milo must have gained access to the boat after Gilbert had been inside the boat. For Milo to have gained access to the boat, the boat must have been locked with Milo's lock, which means Gilbert's work was done. Gilbert has no further need to return to the boat once Milo's lock is back on, and if Milo's lock is not back on, Milo can't get inside to retrieve the book.

So that's the first, and most important, problem. Gilbert would not have had any reason to be anywhere near the boat during the time of the Bloodhounds meeting.

The other problem is that even if Gilbert was there to put the lock back on, why was he still there 3 hours after Milo left for the meeting? It would take all of 30 seconds to switch the locks. And why was he inside the boat? You don't need to go inside the boat to switch the locks. It all makes no sense.

There is a murder of another Bloodhounds member (Rupert Darby) later on in the story. Gilbert tries to make it look like Rupert hanged himself out of guilt over killing Sid. If Gilbert hadn't returned to the boat during the Bloodhounds meeting (which he patently did not need to do), then he wouldn't have killed Sid in the first place, so there would have been no need to kill Rupert either. The entire plot hinges around Gilbert being surprised by Sid inside the boat, even though Gilbert had no reason to be there, no reason to be there for so long, and no reason to be inside.

Gilbert plants the second padlock in Rupert's jacket, so that when the police discover it they will assume it was Rupert that did the padlock trick to gain access to the boat. But when interviewed by police later, Gilbert implies that he does not think the police know how the stamp was planted in the book. If you don't think the police can figure out the padlock trick, what good does it do to plant the padlock on Rupert?

A solution to the main flaw

The story desperately hinges on Gilbert and Sid meeting inside the boat while Milo is being interviewed by police. We either need Gilbert to be returning the padlock during this time (which is the solution the book gives, but which is impossible, else Milo wouldn't have been able to retrieve his book), or we need Gilbert to do the padlock trick a second time.

Maybe Gilbert wants more revenge than simply planting the stamp in the book. Maybe he has another prank in mind, but this one will take a few hours to set up, so he schedules it for when he knows Milo will be at the Bloodhounds meeting. That it takes several hours to carry out explains why Sid catches him in the act. And whatever he does probably either needs to be undetectable, or he needs to undo it before leaving the boat, or it needs to be embarrassing for Milo in some way, to explain why the police never hear about whatever he did.

Or maybe Gilbert is waiting at the boat for Milo to return, so he can say "Haha! Surprise! It was me who planted the stamp in your book and I bet you don't know how I keep getting in to your boat either!" and bask in the glory of having deceived the Bloodhounds who previously belittled him.

With some sort of change along those lines, the rest of the story would make sense.

Conclusion

As written, the plot is totally impossible. Gilbert would have no reason to be at the boat at the time Sid was investigating, because Gilbert must have switched the locks back prior to Milo collecting the book and going to the Bloodhounds meeting.

But apart from the fact that the plot is impossible, I found it an enjoyable book. I like that it is set in Bath. I like that it is set in the 90s. I consider this to be "present day" and "relatable", compared to the typical 1930s locked room in a giant country mansion. I really liked the padlock trick, because it's such an elegant way of gaining access to a seemingly-locked boat.

I plan to read more impossible crime fiction. Hopefully the next one I read will be logically consistent.

The enjoyment from a locked room murder is in trying to work out how the crime was committed, rather than who did it. Generally all of the clues are revealed as the story progresses, and we are encouraged to try to solve the problem on our own before the big reveal.

What follows is all the different ways I could imagine a locked room murder could be carried out. I present the tropes very roughly in order from most to least promising, in my estimation.

1. The room is booby-trapped

The killer (deliberately) left a rigged object, poisoned food or drink, or a deadly animal inside the room, or tampered with the room itself in some way, such that the victim would be killed some time after they locked the door. (If it's not deliberate, see type 8 instead.)

This is my favourite type of locked room murder, and I think it has the most scope for an interesting story.

Examples from Creek include:

• In "Mother Redcap", people die of fright when looking out of a particular window in an inn. It turns out that there are electrified nails in the floor and the victims are being electrocuted.
• In "The Grinning Man", the bath tub tips its occupants into a tank of water once it has enough weight in it.
• In "The Coonskin Cap", the stab vest is rigged to inflate to suffocate the wearer, before deflating again in time to be discovered. (This is a bit different because the booby trap is on the person rather than in the room, see also type 5 since the vest is commanded to inflate by radio control).

2. The victim sustains an injury before locking themselves in the room, and succumbs later

This could be from a physical attack, or they could have been poisoned prior to entering the room.

I think the seminal example is the 1907 French serial Le Mystère de la Chambre Jaune (The Mystery of the Yellow Room), which features in the Creek episode "The Letters of Septimus Noone". In "The Letters of Septimus Noone", The Mystery of the Yellow Room is turned into a stage musical. In a parallel to the stage musical, one of the actresses is stabbed for real, locks herself in her dressing room, and subsequently dies in the locked room.

This is a clever idea, but there just aren't all that many ways to give someone an injury that they will later die from.

3. The victim is still alive when discovered, is killed by the discoverer

The discoverer kills the victim upon "discovering" them, and then declares it to be an impossible crime because the room was locked upon their arrival.

The discoverer should ideally appear implausible as a murderer, or have a method that makes it appear as though the murder took place substantially before the discovery, otherwise the possibility that they did it is quite obvious.

4. The murderer hides inside the room and sneaks out once the coast is clear

Something about the initial investigation of the room is not sufficiently thorough, leaving the murderer the opportunity to remain hidden and then escape later.

For this to be a viable strategy, the killer must know that the initial investigation of the room will not discover them, otherwise it's too risky. Maybe they know it will be someone who will be in shock at the discovery of the body, and will immediately leave the scene to seek help.

5. The murder was committed from outside

The victim locks themselves in a room and the murderer kills them without entering the room, and ideally makes it look like the murder could only have taken place from inside the room.

Maybe a gun is fired through a letterbox. Maybe poisonous gas is piped through an air vent.

6. The murder took place elsewhere and the body was dumped in the room

The room must have some method of entry that you can chuck a dead body through, but which a living person could not use. For example, maybe the room is deep underground and the only doorway has been bricked up for years, but for some reason there is a skylight or something directly above the room that the body could be dropped into.

Probably hard to make a good story out of this, because as soon as we learn about the entry method, it will become clear that a body could have been dumped through it.

7. The method of watching the door is defective or malicious

The security guard falls asleep. The security guard is the murderer. The "witnesses" were in on it. The CCTV is hacked so that it loops innocuous footage of nothing happening, instead of the real footage of the murderer coming and going.

If you're writing a locked room murder and you don't plan to use this trope, you would probably actually lock the room, rather than leave this possibility open, since the murder is so obviously trivial if the "watched room" is not watched properly. So if the room is merely watched, rather than locked, then it's either the case that the watching is defective/malicious, or the writer explicitly wants this to appear as a strong possibility.

8. It's not a murder, it's an accidental death

There is no murderer, the death just happened by accident after the victim had locked themselves in the room.

Maybe the victim banged their head on something, maybe they suffered a heart attack, maybe they didn't know that some innocuous-looking object in the room was actually deadly.

9. The room was actually locked from the outside

The murder took place conventionally, then the murderer left the room and locked the door in such a way that it appeared that the door had been locked from the inside.

10. The room has a secret exit which the murderer used

The murderer goes into the room through the normal entrance, commits the murder, locks the door, and escapes through a secret exit which nobody else knows about. Or maybe the victim was already in the locked room, and the murderer both entered and exited through the same secret passage.

Similarly to type 6, it's hard to make this into a good story, because either the reader knows about the secret passage or they don't. If they know about the secret passage then the solution is trivial, and if they don't know about the secret passage then they don't have a fair chance of solving it.

11. It's not a murder, it's a suicide

The victim locked themselves in the room and then killed themselves.

For this trope to be at all compelling, we probably either need to be looking to explain why the person committed suicide, or it needs to appear impossible as a suicide. Suicide is the default assumption when a dead body is found inside a locked room, so there has to be something else, otherwise we just have "It looked like a suicide, it was a suicide, the end".

In the Creek episode "The Seer of the Sands", a man kills himself (albeit not in a locked room) after receiving a fax containg good news. Why? He misread the fax because a tiny insect had landed on it which appeared to place a comma in the middle of a sentence, changing the meaning completely.

Slotted disc

I modelled a slotted disc to intermittently block the IR light on the light gate. The disc has 36 slots, which means the resulting signal from the light gate will have 72 edges per revolution, so we can detect crank position in 360/72 = 5° increments.

I printed the disc in PC Blend, and fitted it to the crankshaft:

It's just held on with friction, but that's adequate. You can see a video clip of the engine running with the disc in place if you like.

One advantage of measuring the crank position this way is that it adds no friction to the engine. OK, maybe a tiny bit of air resistance. But if you were to connect a standalone rotary encoder you would be adding some friction, which could make a measurable difference to how the engine runs at low speeds under no load (which, after all, is the usual running condition for a model steam engine).

Datalogging

To collect data from the sensor, I wrote the stupidest Arduino program I could think of. The (analogue!) output of the light gate is wired straight to a digital input of the Arduino, and I'm just reading it in a loop with digitalRead() and outputting millis() to the serial port whenever the input changes.

This seems to work well enough, even without any signal conditioning. I was worried that the signal would sometimes flutter around the threshold and some debouncing would be required, but it doesn't look like it.

I do want to try reading the light gate with an interrupt instead of a loop. That should improve the precision of the timestamps, and means I get to find out about changes to the pin state even while the program is busy doing something else (for example, writing to the serial port or reading the pressure sensor).

It turns out that millis() is not really precise enough. At 72 steps per revolution, at 500 rpm we have 600 steps per second, which means there's only 1 or 2 milliseconds between steps, which doesn't give us a lot of precision on the rpm measurement (i.e. it acts like steps are either 1 or 2 ms apart, which would imply either 417 or 833 rpm, with nothing possible in between). So I should switch to micros() instead.

But with a bit of post-processing we can gain some precision by looking at the time taken between multiple consecutive steps (I made my script look at all steps within the last 20 ms). This does cause some unwanted smoothing. Using micros() would be better, I'll do that for next time.

I have this plot showing the engine speed over time. It is labelled with 7 obvious separate "stages". These are caused by me manually adjusting the regulator on the air supply.

And this is the same data again, but a scatter plot of RPM against crank position:

Hopefully that's easy enough to understand. During each revolution of the engine, the time is logged at 72 separate points. From the post-processing mentioned above, we turn the timestamp into the RPM at each of these points, and we know to reset the crank position to 0 every 72 points.

(Stage 4 is noisier because the engine speed varied more, consult the other graph; I think the engine was rattling a bit too much at this speed).

This shows what I really wanted to see, which is firstly that the speed varies throughout the cycle (i.e. the engine speeds up during the power stroke and slows down during the exhaust stroke). But most importantly it shows that the light gate is not substantially losing steps! Because the speeding up and slowing down is aligned extremely well to the 72-step cycle. This is good.

You can see that the peak engine speed is in roughly the same place during each stage, which means over the course of 90 seconds it can't have gained or lost more than a small handful of steps, which means it is probably viable as a crank position sensor.

Index pulse

If the sensor did gain or lose an important amount of steps, the solution I had in mind was to make some sort of "index pulse" that fires once per revolution, always at the same place. You could do this by putting a magnet on the flywheel and sensing it separately, or (easier) break off one of the teeth on the slotted disc and look for the longer pulse from the light gate. You'd then only get 70 steps per revolution, but one of them would be 3x as long.

A big advantage of having an index pulse is that it gives you absolute crank position. Without an index pulse, you know how many degrees the crank has rotated since you started watching, but if you don't know what position it was in when you started then you still don't know where it is at any other time. With an index pulse, you can turn your relative crank position measurements into absolute crank position measurements as soon as you've seen 1 index pulse.

So I think I probably want to add some sort of index pulse, just so that I get more confidence in the absolute crank position. Being able to re-synchronise once per revolution is an added bonus, although not apparently necessary. The big benefit is not having to manually synchronise when the logging starts.

Cylinder pressure sensor

Before I did all the light gate stuff, I did also move the pressure sensor to the top of the cylinder. So now instead of measuring the inlet pressure, I'll get the pressure inside the cylinder.

I did this by putting a threaded hole in the brass part that plugs the top of the cylinder. The sensor is screwed down just far enough that the bottom of the sensor is flush with the bottom of the brass part, so I think it is not substantially affecting the volume inside the cylinder.

I did take some measurements from this and compared to the simulator:

It's not a brilliant match; the simulator predicts a couple of sub-peaks that just don't seem to be present in the real data. Not sure what's going on there. Maybe the sensor (or Arduino ADC) doesn't react fast enough to changes in pressure? Maybe my sampling rate is not high enough? Maybe the simulator is wrong? Not sure.

But at least the speed and peak cylinder pressure sort of line up for the same supply pressure (which we'd expect, given the work I did on loss calibration).

Next

So I probably want to switch to using micros(), break off one tooth of the slotted disc, try out using the light gate to trigger an interrupt pin instead of reading it in a loop, and make it log cylinder pressure and crank position at the same time. Then I can finally try to reproduce the simulator's pressure-volume diagrams, which will be a really good milestone.

I made the light gate using components I already had in the junk box, and it only took a few hours, most of which I spent on trying to get the wrong kind of infrared sensor to work.

(The dark one is the photodiode, the clear one is the LED)

It's just an infrared LED and infrared photodiode facing each other, and a couple of resistors, on a piece of stripboard, inside a 3d-printed housing. The slotted disc will run in the gap between the LED and the photodiode so that it intermittently blocks the light path, which allows me to count the pulses to work out the angle the crank has rotated at any given time.

The mounting holes on the plastic housing will eventually be used to screw the sensor to a piece of material that will hold it in place near the engine.

There is a small amount of strain relief on the cable when the PCB is screwed down, you can see in this CAD screenshot:

As the PCB is screwed down ("up" in the pic), it clamps the cable into the housing.

The IR LED is powered by 5v and has a 510 Ohm current-limiting resistor. The photodiode has 5v applied to the positive side, and the negative side is the "signal" output. The signal is pulled to ground with a 680 KOhm resistor. This signal pin goes to a digital input pin on the Arduino.

At first I thought I was going to have to use an analogue input and manually configure a threshold value to distinguish "light" and "dark" readings, but the voltage difference between light and dark is almost the entire 0v to 5v range, so it works plenty well enough connected to a digital input.

Some of the schematics available online involve using the photodiode to control a transistor to provide 5v to the microcontroller, but I found this not to be necessary. Probably I am in the easy case because my LED is so close to my photodiode; if you were trying to receive instructions from a television remote control from the other side of the room you would get much less signal. The method I ended up using is from an IR Photogate Instructable.

What didn't work was to use one of these "infrared receivers":

These detect infrared sources that switch on and off at 38 kHz, and the output goes low when it detects a 38 kHz signal and high when it doesn't. I thought this would be easier than using a photodiode (even though I need to make a 38 kHz signal instead of a solid light source) as I thought the photodiode would be too analogue and annoying. The issue I found with the "infrared receiver" is that the output seems to go high again after about half a second, even if the 38 kHz signal remains present. I guess these things are intended to be used for brief bursts of communication, which will involve the 38 kHz signal rapidly appearing and disappearing. Using the photodiode turned out to be easier.

From some testing by sticking a piece of paper in the gap between the LED and the photodiode, I think this is going to work nicely.

Measuring RPM

Last time, I measured the engine speed by measuring the time between the pulses in the audio track of the video.

I'm now measuring it with the inlet pressure sensor! This is possible because the pressure drops when the port opens, so you can look at the frequency of the resulting waveform to work out how fast the engine is running:

My first attempt at this was to write some stupid ad-hoc code for peak detection and then work out the engine speed based on the time between peaks. This didn't work very well:

I think the data was too noisy for my stupid peak detection to work properly.

So next I tried taking the FFT of a sliding window of pressure readings and then saying that the frequency with the highest peak (within a sensible range) is the most likely engine speed. It sometimes picks up harmonics though (e.g. 2x or 0.5x the true speed).

I solved this by manually adjusting the "sensible range" at different points in the data until all the harmonics seemed to be gone.

And finally, taking an exponential moving average of this data, we get the RPM plot that I'm treating as the "truth".

Working out RPM from the inlet pressure is good enough for this purpose, but I still want to use an encoder on the crankshaft to measure the crank position when I come to plotting pressure-volume diagrams.

Measuring supply pressure

Last time, I got the supply pressure from the number shown on the gauge on the compressor. I don't have a lot of confidence in that gauge, and now that I have the inlet pressure sensor I can use that instead. I'm saying that the supply pressure at any given time is the peak pressure from the inlet pressure sensor over the last 0.6 seconds. This is a long enough window that it covers a peak from the last cycle even at the lowest engine speeds.

So then we can draw a scatter plot of speed-vs-pressure from every datapoint:

It is a bit noisy, but it does follow the form of the theoretical relationship:

Loss calibration

Armed with a representative sample of speed and pressure datapoints, I manually adjusted the loss torque in the simulator until the simulated engine speed matched the real engine speed for the same supply pressure (this time also using the "reservoir" supply model I described in "More inlet pressure measurements").

Here's the old vs new loss calibration curve:

(I don't know why the new line is wiggly - I'm guessing it's caused by uncertainty in where exactly the "true" curve runs from when I picked samples from the speed-vs-pressure scatter plot).

Because the engine now runs more freely, it can both run slower without stalling and run faster on any given supply pressure. So the range of recorded speeds is larger, and the friction is lower at all speeds.

And here's the quadratic fit from LibreOffice:

In light of this, I've added support for a quadratic loss model to the simulator.

I don't know if that UI is really the best way to represent it. Maybe I should make it clearer that the values are coefficients of a quadratic function.

And here's the scatter plot from above (red), with the measured speed-vs-pressure curve from the simulator (green), using the quadratic loss model:

That's close enough for me!

Engine modifications

Now that the loss model in the simulator might be reasonably close to reality, I wanted to simulate some engine modifications to see if we can find improvements on the standard Wig-Wag configuration. All of these cases are with 200 kPa supply pressure (= ~30 psi).

(Note all the following are using a simulated version of my actual engine, which means the bore is 15.2mm, the gap above piston at TDC is 8mm, and the piston+rod length is 61mm).

Firstly, if we were to drill out the cylinder port, would we be able to improve efficiency, power, or both?

The standard cylinder port is 2mm. We can improve both peak power (by about 50%!) and peak efficiency by drilling this out to maybe 4mm. (Though note that peak power and peak efficiency are not achieved at the same speed).

What about if we left the cylinder port as standard and drill out the inlet/exhaust ports instead?

The standard inlet/exhaust ports are 2.5mm. We can improve both peak power and peak efficiency by drilling these out to about 4.5mm.

And finally, what if we drilled all 3 (to a single size)?

Somewhere between about 3mm and 3.5mm looks best.

So it should be pretty easy to get about 50% more power out of a Wig-Wag at 200 kPa just by tactically drilling out some/all of the ports. I don't want to make any modifications to my actual engine until after I have managed to plot some pressure-volume diagrams from it, so my next move is probably to put a threaded hole in the top of the cylinder and fit the pressure sensor to the cylinder, and make the crank position sensor.

Concept

Imagine one of those lawn sprinklers that has angled nozzles that spray jets of water as they spin around in circles:

Now imagine the arms are made longer, and the jets are more powerful, and the angles and motion are controlled so that the jets of water land exactly where you want them.

In fact, you don't have to imagine, because I made a simulation.

Check out my simulation here

It's not super easy to control (the angle controls, in particular, are confusing) but you can make the jets loop around and return to the nozzle they came from, or all cross in the middle, or do several spirals before returning to the ground, etc.

First implementation

I don't know if I'm ever going to actually build this, but for a proof of concept we could mount the Coriolis fountain in a small paddling pool.

We'd put a raised platform in the centre with a large gear on it, fixed in place. There would be a frame on top of this platform which is able to rotate around the centre. The frame would be driven around the gear by a motor mounted on the rotating frame, something like this, from my robotic telescope mount:

The speed of the motor would be radio-controlled. Obviously it needs to be waterproof.

The rotation axis would be hollow, and a submersible pump would go down it to be submerged in the water in the paddling pool. It doesn't matter that the pump rotates with the frame, it still pumps. The output of the pump splits into 3 (or however many) and runs off down the arms to the nozzles. The nozzles have 2-axis control via radio-controlled servos.

Then we just need to put a battery and an RC receiver on the rotating frame, and the whole thing can be controlled remotely with normal RC gear. Since both the pump and the drive motor are located on the rotating frame, none of the cables or hoses need to pass through rotating fittings.

We could add per-nozzle flow rate control by having a (strong?) servo motor clamp down on the hose to limit the flow.

More

Once the concept is proven with manual radio control, we can look at automating the control. In increasing order of complexity:

1. the rotation speed and nozzle angles could be set manually and left alone
2. we could have a small number of preset patterns and the system cycles between them either on a timer or on command
3. we could have animated patterns, where the nozzle angles or rotation speed can vary over time

Probably we'd want a microcontroller onboard the rotating frame to control the animations. And we'd either want some power cables running to the frame through slip rings, or maybe solar panels if we can tolerate it not running when it's not sunny.

And it would be cool to make it a permanent installation in a big pond or something. Sadly I don't have anything suitable. Do you have a big pond or lake? Do you want a Coriolis fountain? Email me, let's talk. I would love to help you make a Coriolis fountain.

The short review is that I like it and if you want to build a regenerative receiver you should get it.

It took me about 3 hours to put the kit together. The instructions are pretty good and easy enough to follow, they come in a nice booklet.

Regeneration board

The first job is to solder up the "regenerative board". I was surprised to find that the copper side of the PCB is isolation-routed.

I have had a go at this myself in the past:

And while that particular PCB did work, it is the only success I've had. Every other attempt has failed in one way or another, because I was trying to design to the kind of tolerances I expect from JLCPCB.

I think the method used in the KRC-2 design is much better, because it maximises the width of the traces and the area for the solder pads. You are much more likely to produce a successful PCB with this method than if you try to lay it out like a normal PCB. I wonder how much trouble it is to make layouts like this in KiCad...

The other side seems to be screen-printed:

The footprint for the transistor requires the "base" leg to be bent the opposite way to what you'd normally expect, which confused me for a few minutes:

It's bent towards the flat side instead of the rounded side. This is contrary to both the usual practice and the diagram in the instructions:

The coil seems to be wound around a piece of plastic M10 threaded rod:

I think this is a good idea, as it gives you nice consistent spacing between the turns and means you don't need insulated wire.

Case assembly

With the PCB done, we move on to putting everything together in the case.

I was surprised to find a second PCB already fitted inside the case:

This is the "oscillator board". Why is this ready-made? Why not let me put this one together myself?

Probably because the soldering is much trickier (it uses SMD resistors, and has quite a few flying parts connected to the rotary switch) and by the looks of it the variable capacitor needs to be trimmed by hand, which presumably is quite a skilled job and can't be done until the board is assembled. Fair enough.

The KRC-2 seems to be an unusual regenerative receiver, in that it uses a 10.7 MHz oscillator mixed with the incoming signal from the antenna, which I haven't seen on any other designs. According to the documentation, this reduces spurious emissions and makes the regeneration control more stable. I don't know enough about radio to judge the pros and cons.

Assembling the case involves attaching the battery holders to the back panel, the "sense" and "regen" knobs and speaker to the front panel, and wiring the battery, antenna connection, regenerative board and oscillator board all together.

The speaker is glued to the back of the front panel with PVA glue around the edge of the paper cone. This would not be my first choice, although this particular speaker doesn't appear to have any provision for mounting hardware. Maybe you could hold the back of the speaker down with tabs against the metal? The glue seems to have stuck well enough, but I'm not sure whether it would rip the paper cone if I ever tried to take it apart.

The last step of the assembly is to solder wires on to the solder tabs at the top of the oscillator board. The instructions ask you to take the top panel off the case in order to access this, but I found this to be impossible. You can see here that the top panel is glued to the batten, which is also glued to the side panel:

I'm not even sure how this is meant to have gone together. If the top panel were removable when the screws were removed, then it would either take the batten with it or not, and in either case there would be nothing remaining to join the side panel to the top panel. I wonder if this is a design revision that hasn't made it into the instructions?

Anyway, unable to remove the top panel, I came up with an alternative method of soldering the wires to the tabs, which was to use some tweezer heroics to hold the wires in place and then reach the soldering iron all the way in to the corner:

This worked, but after I finished I had a better idea: I think if you were to remove both the front and rear panel, you could solder the wires in place on the bench at your leisure, and then snake one of the panels through the housing and screw them both back on from the outside.

Trimming

To trim the regeneration, there is a 20-turn trimpot on the regenerative board. The instructions ask you to set "sense" and "regen" to the middle, turn the trimpot all the way down, switch to "range 1" (which I assume is the 1-10 MHz range), give it 10 seconds to warm up, and then slowly advance the trimpot until a hiss is heard in the speaker.

Well I heard a hiss in the speaker even with the trimpot at minimum, and it hardly changed at all even when I turned it all the way to maximum.

I thought that probably I had made a soldering error, but before taking it all back apart I played around with it a bit and worked out that the quiet hiss I had been hearing in the speaker "doesn't count". The real hiss that you're looking for is much louder, and you're more likely to find it if you tune somewhere to the middle of the frequency range instead of at the very bottom.

Use

Moving left to right:

The "regen" knob is a potentiometer on the regenerative feedback, to control the strength of the regeneration. If you set it too low you don't hear anything (beyond the quiet hiss that is always present). Once you approach the critical point the hiss gets louder, and beyond the critical point it causes whistles and buzzes. You can tune it just beyond the critical point if you want to listen to Morse code or single-sideband, and just below the critical point if you want to listen to AM.

The "sense" knob is a potentiometer on the antenna input. It is suggested that you use this knob to control the audio volume, but it seems like the receiver would benefit from a second volume knob because attenuating the incoming signal does more than just make it quieter.

The next knob is labelled "fine" and is used to make fine adjustments to the frequency after having selected a frequency with the large knob.

The last knob is a rotary switch that selects between the 3 different frequency ranges, and is also a power switch.

The receiver is quite hard to tune, but that's fine, that's what makes it fun to use! If I just wanted to switch a radio on and listen to something I would have bought a mass-produced one.

It seems to be sensitive to how close my hand is to the left hand side of the box, which is where the coil on the regneration board lives. In some configurations you can almost play it like a Theremin.

I expected to be able to tune some medium-wave stations at the bottom end of the 1-10 MHz range, but was almost entirely unsuccessful. I did manage to find TalkSport once, but it was extremely sensitive to small movements on all of the knobs. I've had most success listening to international stations between 10 MHz and 15 MHz. I don't know how much of this is due to my antenna (I have ~5 metres of wire hanging indoors, which is about 1/4 of a wavelength at 14MHz), how much is due to the receiver, and how much is due to ionospheric conditions. Sadly the foreign stations mostly seem to speak foreign languages, so I struggle to understand what they're saying.

I tried grounding the receiver through the electrical earth, but it seemed to introduce a lot of interference (choppy noises). For now I've got the "ground" connection tied to a metal shelving rack, which seems to be a slight improvement over nothing at all. I intend to replace the antenna with a wire hanging in the roof ridge in the loft, with a coax running down to my office. I don't know what to do about grounding, if anything. The KRC-2 instructions include a diagram that suggests hammering a spike into the ground.

This came about because I tried to reproduce the real-life observations in the simulator and did not succeed, so I repeated the real-life measurement and this time I got a different result. (As any good scientist knows: if you don't get the results you want, you should repeat the experiment until you do).

This chart shows today's measurement of inlet pressure over time, for representative samples with 4 different supply pressures:

The "0" kPa line is from running with a 0 indicated on the regulator pressure gauge (video), which maybe is about 30 kPa judging by the line on the graph?

And the trace from before was more like this:

That's with 200 kPa supply pressure. If we draw straight lines past the sharp dips, it looks more like the new version:

So my best guess at what has happened is that the engine previously had a leak from the port faces at some point during the cycle, which is now fixed. Such a leak would allow air to rapidly vent to the atmosphere through the inlet port, instead of having to pass through the cylinder.

In between, I had dismantled the engine, cleaned all the dirty oil off, re-lubricated and reassembled. I don't know that that would have made a substantial difference, but I can't think what else I changed.

The other obvious difference is that the peak pressure now approximately matches the indicated supply pressure instead of exceeding it. I also don't know why this would have changed, but it could just be that the gauge on the compressor is not very accurate.

Simulator

I added an option to the simulator to allow the configuration of a "reservoir" which supplies the cylinder, to allow the pressure to drop when the port opens and then slowly fill back up once it closes. You can use it if you turn off the "Supply from infinite volume?" option.

It basically acts like a spherical air tank of a given volume, with an orifice connected to the inlet port on one side and an orifice connected to an infinite volume of the nominal supply pressure on the other side.

The reservoir configuration of a 12000 mm³ volume and a 0.7mm fill port gives pressure drops roughly matching what I observed in real life.

I don't yet have a convenient way to overlay the plots on the same chart, but here are traces of the reservoir pressure from the simulator for 200 kPa, 140 kPa, and 70 kPa supply pressures, with the load manually adjusted as per the calibrated losses so that it runs at the same speed as the real-life engine did at the same pressure.

The range between the maximum and minimum value is shown in the number at the left hand side. Comparing this to the real-life chart from above (shown again for your convenience):

We can see that the magnitude of the pressure drop is about the same in all cases, although the waveform is not quite the same.

With 200 kPa supply, the pressure drops to about 150 kPa. With 140 kPa supply it drops to about 105 kPa, and with 70 kPa supply it drops to about 55 kPa.

The shape of the falling edge of the reservoir pressure plot kind of mirrors the inlet port flow rate (as you'd expect):

I don't know why we don't see a shape like this in real life.

Possible explanations for the differing waveform include that the hose in real life is not actually a spherical reservoir, or that the regulator is actually supplying the hose with high pressure air through a variable-sized orifice, whereas the simulator has a fixed-size orifice supplying lower pressure.

But I'm happy that the magnitude of the pressure drop lines up with real life.

Next

I'll probably want to re-record the loss calibration data now that the engine is running better (the fact that it can now run on lower pressure suggests that losses are reduced), and re-calculate the calibration curve in the simulator now that I can use the "reservoir" model to simulate the pressure drop in the supply hose.

I also still want to try adding a small reservoir to the real-life engine, to see if it reduces the magnitude of the real-life pressure drop.

The idea is that the hose restricts the flow of air. So when the engine's inlet port opens, the air at the end of the hose drains into the cylinder, and it takes time for the hose to fill back up again.

Pressure sensor

My setup looks something like this:

In that picture, the laptop is running the Arduino IDE's "Serial Plotter", there is an Arduino Nano on the breadboard running the "AnalogReadSerial" example program, there is a pressure sensor connected to an analogue input of the Arduino, and screwed into an aluminium fitting that I made, and the aluminium fitting is inline with the air supply hose.

This picture better shows the connection of the sensor to the air hose:

I got the sensor from eBay for about £15 (search "pressure transducer"). It takes a 5 volt supply and gives an analogue output between 0.5v and 4.5v, supposedly linear in its range of measurement. Mine has a 30psi range (= 200 kPa), but there are others with different ranges.

First measurements

Here's a video of the system in use:

You can clearly see that the measured pressure is not constant, and does indeed vary throughout the engine's cycle. The simulator does not simulate this, it assumes that the inlet pressure is constant. The simulator acts as if the inlet port is connected to a high-pressure "second atmosphere".

There are a few points in the video where the plotted pressure goes up and the engine slows down: this is because I was pressing my finger against the flywheel to add some load to the engine. It's not obvious to me whether the measured pressure increase is primarily because the engine is running more slowly (so there is more time for the hose to "fill up" with air when the port is closed), or because the engine is providing more resistance (so the hose doesn't drain as quickly when the port is open). Maybe both.

The pressure measurement has two obvious peaks throughout the cycle, and the smaller one gets larger as the load increases:

From playing around, it seems like it stalls at the point where the small peak meets the large peak. Not sure if that means anything interesting.

The long hose

To test the hypothesis that the long hose is causing the restriction, I can remove it and connect the air gun directly to the compressor:

If the long hose is the restriction, then when the hose is connected we'd expect the measured pressure to be lower and to fluctuate more.

I captured pressure data from several runs at three different pressures, both with and without the long red hose.

The "error bars" are showing the range between the minimum and maximum recorded pressure from the sensor, over a period of 5 seconds in each case.

We can see that with the long hose connected, the average pressure is lower as we expected, but also the magnitude of the fluctuation is lower. So I think that while the long hose does increase flow restriction overall, it also acts a bit like a second air tank, allowing more air to flow for short periods without changing the pressure as much, because it can drain out of the hose without having to come through the valve on the main tank.

It may also be notable that the maximum recorded pressure appears to exceed the supply pressure in each case. I am not yet sure whether this is because of the engine pushing air back into the hose, or just a result of mis-calibration of either the electronic sensor or the gauge on the compressor.

Conclusion

For a more accurate simulation: instead of exposing the inlet port directly to the "second atmosphere", the inlet port could be exposed to a reservoir (representing the hose) that is itself connected to the second atmosphere through a small orifice. We'd then use the selected air flow method to calculate both the flow from the reservoir to the port, and the flow from the supply pressure into the reservoir. I'd also like to add an "oscilloscope" to the simulator so that you can get it to plot things like the pressure in the reservoir over time, to see how it compares to the real-life data.

For a better-running engine: maybe add a large air reservoir near the inlet port? It should make the engine run faster on any given supply pressure because the pressure won't drop as far when the port is open. It might be interesting to try this with a valve on the entrance to the reservoir, so that it can be "turned off" while the engine is running, to more clearly see what effect it has.

I've also observed that the engine seems to be running in nicely. Last time it wouldn't run below about 10 psi, but today I've noticed that it will run even at pressures low enough that the gauge reads 0.

Electricity analogy

I'm kind of getting into analogue electronics at the moment, and it's surprising how many ideas can carry over.

We can think of pressure as voltage and flow rate as current.

Restrictions to flow rate are the same as resistance. Reservoirs for air are a bit like capacitors (e.g. with one leg attached to ground: they can store up pressure and release it later).

The engine presents an inductive load, just like an electric motor, because of the inertia of the flywheel (it wants to keep pulling air through even once the supply is switched off).

The supply hose acts like a wire with high resistance, and combined with the resistance presented by the motor, at any given point in its cycle, they form a voltage divider, which we are measuring with the pressure sensor (i.e. voltmeter).

So then the issue with the pressure dropping when the port opens is just like brownout when a circuit increases its current draw. The solution to brownout in electronics is to put a bypass capacitor near the power input to the affected device, and the solution I have proposed for the engine is exactly the same: putting a "reservoir" near the inlet to the engine is the same as adding a bypass capacitor.

For a better analogy to a capacitor, we'd probably want to make the reservoir out of a balloon. That way it is actually acting against the atmosphere (i.e. ground potential) instead of just getting pressurised on its own. What we've got with a reservoir is basically like a capacitor that only has one terminal. Like a big metal plate hanging off a wire. I'm not actually sure why a capacitor works more than twice as well as a single metal plate does. I should find out.

You may recall that my engine was not working very well because the port faces were not sitting flat. My Dad suggested an easy fix would be to enlarge the pivot hole so that the spring is able to pull the faces together. That's easier than anything else I had in mind, so I have done it. I only drilled it out by 0.2mm. You can still hear air leaks when the engine stalls, which suggests there is still not a perfect seal, but that is probably unavoidable.

1. Gathering data

Methodology

I set up a video camera pointing at the pressure gauge on the compressor, and within earshot of the engine. Using the view of the pressure gauge I can tell what pressure is being supplied to the engine, and by counting pulses on the audio track I can tell what speed the engine is running at. I adjusted the regulator to change the supplied pressure, and allowed the engine to run for a few seconds to stabilise before adjusting the regulator again.

I then watched the video and noted down timestamps at which the engine speed was stable, along with the corresponding pressure at that time. I then scrolled through the audio track in Audacity and counted the pulses.

The highlighted portion corresponds to 5 revolutions. Audacity says the selection spans 512ms. 5 revolutions in 0.512 seconds = 9.77/sec = 586 rpm

My data is all in a LibreOffice spreadsheet in case you want to look at it. And the source video is on YouTube.

My Wig-Wag is not built exactly to the drawings. The bore is 15.2 mm (+0.2mm vs spec), the length from the crank pin to the top of the piston is 61mm (-2mm vs spec), and the gap above piston at top dead centre is 8mm (+3.25mm vs spec, I am as confused as you about where the extra 1.25mm has come from).

Confounders

That some air leaks out when the engine is stalled implies that the port face is not a perfect seal. This also implies that some air leaks out while the engine is not stalled (but you can't hear it over the noise). But the simulation assumes the air path is perfectly sealed. Maybe the simulation should gain a parameter to configure the diameter of a "leak hole" in the top of the cylinder, to simulate the leak from the port faces.

It is likely that when the inlet port is open, the actual available pressure drops, but this is not reflected in the gauge because of the length of hose in between the gauge and the engine. To find out, we should put a pressure gauge closer to the engine.

The lowest marked point above 0 on the pressure gauge dial is 10 psi (~69 kPa), and the scale is very obviously linear everywhere except at the lower end, where there is not much of a gap between 0 and 10 psi. This makes me suspect that the scale is not very accurate at the lower end, so the data I have recorded is likely to be less accurate at lower pressures than at higher pressures.

Results

Evaluation

For a sanity check, we can compare the qualitative shape of the curve to pressure-speed curves generated from the simulator, and find that it matches pretty well: as you increase the pressure more and more, the engine speeds up less and less (diminishing returns). Also, the curve intersects the x axis and not the y axis, which means there is a minimum pressure required for the engine to run, and it will not run at 0 pressure.

2. Calibrating losses

Methodology

For each of the air flow models in the simulator (TLV, Bill Hall, Trident 1, Trident 2, Bernoulli, Linear), and for supply pressures at 25 kPa intervals from 75 up to 275, I adjusted the simulated friction losses of the engine until the simulated engine was running at the same speed as the real engine.

Under the assumption everything else is accurate, we have thus determined the amount of friction losses that the real engine was suffering from at this speed.

Results

Evaluation

Aside from TLV and Linear, all of the other air flow models have regions where increasing the engine speed actually decreases the amount of friction. This is intuitively tough to swallow, although you might be able to argue that some parts of the engine are transitioning from stick-slip to sliding friction (maybe the port faces?).

I think this is not the reason. I think the problem is simply that those air flow models are too pessimistic about how quickly the air can flow in/out of the engine, so the only way to make the speed match as pressure increases is to let them have less friction. The loss curves for "TLV" (from TLV) and "Linear" (my uninformed guess at how air flow works) look much more believable to me. Since the TLV model is from "A Steam Specialist Company", and the Linear model is pulled out of my bum, I'm going to say the TLV model is probably the better one.

Currently the simulator only accepts a linear loss model (i.e. a static amount of loss, plus a loss that rises proportional to engine speed). I tried fitting a quadratic curve to the loss data from the TLV model. It doesn't look exactly the right shape, but it is better than a straight line:

3. Conclusion

The next things to try are:

1. run the same measurement again but with a second pressure gauge, closer to the engine, to see how much the supply pressure actually drops when the port opens
2. add a "leak hole" to the simulator and see if adjusting the size of the leak hole can make the simulated engine better match the real one
3. make the simulator accept a quadratic loss model, and configure it according to the discovered curve above
4. absent any indication to the contrary: eventually remove all of the air flow models except for TLV

I made it with my mini lathe and CNC mini mill.

I'm running it on compressed air rather than steam, purely for convenience. I'm more interested in the engine than the steam for the time being (although I am a bit interested in the steam, so maybe I will make a steam boiler one day).

I managed to fit the "pivot rod" to the cylinder out-of-square, which means the port face on the cylinder does not sit flat against the port face on the main column. This is very bad because it means the port faces only make contact in one corner, which leads to premature wear, and because the port face doesn't seal very well, which wastes a load of air and reduces efficiency.

I did try to run the engine with the spring tensioner done up quite tightly to try and make it wear in, but this was a bad idea and I have caused the two faces to gouge each other in the corner where they contact. Bad.

I believe the cause of the problem is that I didn't take enough care when tapping the female thread in the cylinder.

The solution will be to take the pivot rod out and either make the thread a very loose fit and glue it in with Loctite such that it is square, or re-machine the port face on the cylinder so that it is perpendicular to the threaded hole, or a bit of both.

Once I have made the engine run the way it should, the next step is to fit a pressure sensor to the top of the cylinder, and some form of crank position sensor to the crankshaft, and plot pressure-volume diagrams from real life to validate the accuracy of the simulation.

I have a video clip of my engine here:

Knurling

The drawings call for the spring tensioner to have a knurled grip on it, but I do not have any knurling tools. My solution was to use my homemade 4th axis to cut a diamond knurl pattern using a 90-degree V-pointed cutting bit, and that worked pretty well:

I have a video clip of that process too. It is not a fast operation.

Simulator

There have also been quite a number of improvements to the simulator since last time, most notably it now supports double-acting engines and it can now plot torque curves and pressure-speed curves, for example:

You can play with my simulator if you want.

You might enjoy trying to find an engine with the smallest displacement that can provide at least 10 Watts of power at 50 kPa inlet pressure. You'll need to adjust the "load" field to get some useful work out (otherwise the engine will accelerate up to the point that the work done is cancelled out by the losses), but leave the "loss" and "air flow method" fields alone if you want to compare apples to apples. The rest of the parameters are fair game. If you have a go, send me a screenshot of your parameters, and if more than one person bothers then I'll publish a league table next time.

The top group of parameters configures the engine. The bottom group of parameters configures the environment and simulation.

The diagram of the engine includes a representation of the air inside the cylinder, with the colour representing its pressure. The colours drawn on the flywheel indicate the port timing (red means the inlet port is open, blue means the exhaust port is open, and a fatter band means it's open wider).

The plot on the right-hand side of the diagram is a pressure-volume diagram.

It could do with a bit more explanatory text and maybe a better UI, and probably a way to share a link to a set of parameters. And probably some credit to the people whose engine designs & data I have used.

How it works

The rendering is done with p5.js, which is only marginally simpler than using the canvas directly.

The simulation is completely independent of the rendering, which will become important later when I get around to using simulated annealing to produce optimised engines. I hope to be able to improve on many of the published designs.

I'm actually simulating the engine running on compressed air rather than steam. The difference is that with compressed air you don't need to worry (as much!) about the effects of temperature change and condensation, so the simulation assumes the air remains at room temperature.

At every time step, it computes the mass of air that flows through the inlet and exhaust ports (which can be 0, for example when they're closed, or when the pressure in the cylinder matches the pressure on the other side of the port). It computes the volume in the cylinder based on the position of the piston. With the mass of air in the cylinder and the volume known, the pressure can be computed. Knowing the pressure in the cylinder and the cross-sectional area of the bore, you know the force on the piston. Knowing the force on the piston and the effective acting radius of this force on the crank, you know the torque on the crank. Knowing the torque on the crank and the moment of inertia of the flywheel, you can find the angular acceleration of the flywheel, which gives you an updated angular velocity. With the new angular velocity, you can work out how far the crank has rotated during this time step, which also gives you an updated piston position, and then you're ready to start again at the next time step.

There are only 2 real complications: computing the air flow rate, and calibrating the losses.

Air flow rate

I am modelling the ports as a small orifice connecting the cylinder to an infinite volume of either inlet pressure (for the inlet port) or atmospheric pressure (exhaust port). The cross-sectional area of this orifice depends on the area of overlap between the cylinder port and the inlet/exhaust port, and can be further reduced if the piston partially or completely blocks it off.

In the following image the inlet port (left hand one) partially overlaps the cylinder port, but the port is completely blocked off by the piston so there is no air flow. The flywheel is rotating clockwise, which means the piston is travelling down, which means the port is about to be uncovered, as you can see from the approaching red area on the port timing diagram shown on the flywheel.

This orifice-with-infinite-volumes model is not perfectly right, because even if an infinite volume of air were available on each side, the port itself has nonzero length. It's more like a small pipe connecting the cylinder to the infinite volume. It might be interesting to explore how the results from the simulation would change if the port length were taken into account, but I haven't attempted to do so yet. I'm saying that if your inlet suffers a significant pressure drop when air starts flowing through it, then that is an engine construction problem outside the scope of the simulator.

But we still have the problem of actually calculating what the air flow through the orifice should be.

Linear

My first guess was that the flow rate is simply linear in the pressure difference and the cross-sectional area, and then I'd manually find a fudge factor that makes the performance look believable. This is the "Linear" option in the simulator. It actually works surprisingly well, because as long as the engine is running slow enough, the cylinder pressure adapts to the port pressure very quickly, and then there's no flow rate. So as long as your flow rate calculation has high flow at high pressure differences and low flow at low differences, it'll probably work reasonably well. But it's possible to do better, so I sought improvements.

Bernoulli's Equation

My next attempt was from this Stack Exchange answer which says you can use Bernoulli's equation, which (rearranged) says that the velocity of the air through the orifice is proportional to the square root of the pressure difference.

I'm not sure how reasonable this is because every other source on Bernoulli's Equation that I have looked at talks about flows of liquid, not gas. I don't know enough about physics to judge. I also read that air flow can be "choked" when the velocity meets the speed of sound, which means any further change in the pressure difference can not increase the flow rate of the air. I hacked in a hard cut-off at the speed of sound, but it seems unlikely that the change in flow velocity truly stops abruptly once it reaches the speed of sound, so this is also probably wrong.

Trident

Trident Compressed Air Ltd. has a helpful table showing the flow rate of air through orifices of different sizes, and at different pressures.

I assume the data from the table is accurate since they are an engineering company with "Compressed Air" literally in the name. Sadly they don't provide any formulae, so it's hard to extrapolate beyond the range of pressures and orifice sizes given.

I had 2 attempts at manually curve-fitting this data (these are the "Trident 1" and "Trident 2" air flow methods). They are mostly very similar to each other but diverge quite a lot once the pressure difference drops below the minimum from Trident's table.

TLV

My final air flow method comes from TLV's Air Flow Rate through an Orifice calculator.

TLV is apparently a "Steam Specialist Company" (and not of the video game variety) so like Trident they are also likely to have accurate numbers, but unlike Trident they show their working, so once I worked out what all the parameters meant I was able to lift TLV's formulae straight into the simulation.

They also have 2 different formulae based on whether or not the ratio of the pressure difference to the primary pressure exceeds the product of the specific heat ratio factor and the pressure differential ratio factor (whatever that all means), which I think accounts for the flow rate trailing off before reaching the speed of sound.

At some point I will probably remove the "air flow method" option and make it always use TLV.

Losses

I have 2 sources of data on losses.

I emailed Steve Bodiley about the simulator project, to inquire about the piston on his Simple Oscillating Engine covering up the ports as it crosses top dead centre, and he has kindly sent me a spreadsheet of torque measurements from running his engine connected up to a torque brake, which tell me the supply pressure, engine rpm, and torque load for a handful of different conditions.

Secondly, I found this video of a Wig-Wag running with a pressure gauge visible. Given the supply pressure (and finding the engine speed from counting the pulses in the audio track), I can set up the same conditions in the simulator and work out how much load is required in the engine to produce the observed running speed (rpm).

So far the best I've come up with is to say that there is a fixed loss of 0.001 Nm of torque, plus a speed-dependent loss of 2.5*10^-5 Nm per rpm (i.e. friction losses increase as speed increases).

This model doesn't fit the observations particularly well, and it is probably the weakest part of the simulator. It would probably benefit both from more data and a better model.

Different types of steam engine

You could simulate a multi-cylinder oscillating engine by simply adding up the torque from each cylinder before applying the update to the crank position.

It would be relatively straightforward to extend the simulator to handle other types of steam engine. I'd need to take out the part that calculates port areas and replace it with something specific to the valving of the different type of engine, and the rest can stay the same.

The stand I have built is a "spanning beam" stand, which means the lathe is bolted to a beam which is only supported at the ends. Supporting the beam in the middle is supposedly to be avoided, because if supported in the middle it is possible for the beam to be bent by changes in the load distribution, whereas 2 points always describe a straight line. However for my purposes I think it wouldn't make any difference.

There is a good page about making such a stand, along with several examples: Make a spanning beam metal lathe stand easily.

A while ago I removed some old gates and kept them around for the steel tubing:

I prefer using box section because then you don't have to notch the tubes, but it's hard to beat free.

Spend a few hours cutting the welds off, grinding them back, stripping the paint with a wire wheel, and cleaning up the big dusty mess of rust and bits of old paint, and you have a collection of usable tubes:

I was surprised to find that some of the tubes had been galvanised before they were painted and some had not, also some of them had welded joints in the middle. Were these gates DIY'd by someone who had themselves rescued the tubes from something else? I wonder what previous life they had. I wonder what future lives they may have.

I didn't take any pictures of making the actual beam. As you can see, it is a piece of rectangular box section with some thick lugs welded into it. (This was not part of the steel gates, it is steel I bought for the racing mower front axle but ended up not using).

I laid out a "jig" for welding the end supports on a piece of MDF:

This worked quite well, and I got a nice matching pair of supports:

Welding the rest of it together in thin air was a bit more Heath Robinson, but eventually I arrived at this:

There are nuts welded into the bottom of each tube, to provide for the leveling feet.

It is painted in silver Hammerite paint:

The paint is meant to have a hammered effect, but I would describe the effect as "subtle".

And here is the stand installed, with the lathe bolted to the top:

My garage floor is not very level (compare the adjustment of the leveling bolt on the rear left corner to the front right corner).

Around this time Emma and I tore down an old shed, and I managed to salvage a couple of big sheets of OSB, from which I made shelves for the lathe stand. I prefer plywood, but again it's hard to beat free.

The idea is that the bottom shelf would just be used for storage, and the top shelf would be used as a working surface, and would likely collect lots of swarf. To make the OSB more palatable as a working surface, and to aid sweeping of swarf, I decided to seal the surface with a healthy coat of PVA:

I came back out the next day, ready to fit my shelf, but it still wasn't dry:

I suggest if you are going to do similar you should probably dilute the PVA with water. I left it by a radiator for another day and it eventually dried to my satisfaction.

The top shelf is mounted with screws through the steel tubes. The bottom shelf just sits there under gravity. Since the OSB is a bit too thin to be used unsupported, I have doubled up on the material to stiffen it up, with the top layer spanning across the steel tubes, and the bottom layer sitting between them, like this:

With the shelves installed, the first lathe job was to make some nylon bungs to block up the tops of the exposed tubes:

And finally an Ikea "Helmer" drawer cabinet screwed down to the lower shelf to house the various lathe paraphernalia:

(Yes, I deliberately spaced the horizontal tubes exactly far enough apart to fit a Helmer cabinet. I really like the Helmer cabinets. I have 4.)

I'm still working on the first project in the lathework book (it's a "mini surface gauge", you can see a drawing of one of the parts on the wall behind the lathe), and so far I have found using a dedicated stand to be much more convenient than dragging the lathe out and plonking it on the workbench, although I can't say I've noticed any performance improvement from the increased rigidity. But the second project in the book involves measuring and correcting the twist in the bed so that it turns accurately parallel, which is when my lathe stand will really prove its worth.

Mandatory move to 00003

There is a relatively straightforward argument that the first moves must be 00001 -> 00002 -> 00003. Consider:

1. To turn the last digit into a 3 it has to come from either 2 or 4.
2. The only prime ending in 2 or 4 is 00002.
3. Therefore the only way to get a 3 in the final position is 00002 -> 00003.

This cuts down the space of prime numbers we need to search quite considerably, because we only need to consider the ones that end in a 3 (which is only 2402 of the 9592 primes that fit in 5 digits).

A map

Here's a map of the entire search space, generated with graphviz:

(Click to view large SVG - I suggest zoom out to a level that fits the entire thing on the screen at once)

00001 is at the top and 10723 is the lowest point on the right-hand side.

(I've omitted the arrows that point from every node to 00001.)

The puzzle about the combination lock is isomorphic to pathfinding from 00001 to 10723 on this graph.

I think the graph is small enough (and is laid out sensibly enough) that manual pathfinding is doable, however I think creating the graph by hand would be too much trouble, so I don't think the puzzle is readily solvable by hand in a sensible amount of time.

Mandatory move to 00013?

We already have a proof that it is mandatory to pass through 00003. From looking at the map it is easy to see that it is also mandatory to pass through 00013 if you want to access the subgraph that leads to 10723.

Is there a reasonably straightforward argument that the route must pass through 00013 (along the lines of the argument about 00003) or is it just a very lengthy process of elimination?

Why start on 00001?

I originally wanted to start the combination on 00000, but it took me far too long to realise that there are no legal moves. 00001 is the next obvious choice.

It is kind of unpleasant that 00001 is the sole reachable state that is not prime, but if the start state were 00002 instead then the entire left half of the search space (as shown in the map) would be unreachable. I prefer it this way.

Adventure game

We could imagine using the map to create a text adventure. The simplest way is to just make an ordinary game and connect up the map of the game in the same way as the graph of Prime Combination. Compare to A treasury of Zork maps.

But we can view the digits of the nodes as integer coordinates in a 10x10x10x10x10 5-dimensional hypercube (or, a 5-dimensional hypertorus, since the edges wrap) and then the game is maybe to work out the general rule that decides which movements are legal?

Or maybe you spawn at a random place in the map and you have to work out where you are, without knowing your coordinates, based only on moving around and noting which passageways are open?

Big map

I have a much bigger map of the entire space of prime connections. It includes the isolated subgraphs that can't be reached from 00001. It's at https://incoherency.co.uk/interest/prime-combination-big.svg (warning: 5 megabyte SVG, may lag your comp - I suggest zoom out to a sensible level and then scroll horizontally).

This map contains 2735 separate connected components of 2 or more nodes. There are also 8402 individual primes not shown, which aren't connected to any others. Of the 2735 components, 450 only contain 2 nodes. There are 172 that have more than 10 nodes.

• The input wheels are arranged in the usual way: 5 positions with digits 0 to 9, decrementing from 0 gets you to 9, incrementing from 9 gets you to 0.
• There is a mechanism that prevents you from turning more than 1 wheel at a time.
• If you try to park a digit halfway between 2 values it springs back to one of the values before you are able to change another digit.

I have posted a solution and some analysis.

This is quite reminiscent of the "Centre of mass of a soda can" problem, which asks how the centre of mass of a can of drink changes as the drink is consumed. (You might enjoy giving it some thought if you haven't seen the problem before).

Wall thickness

We want the outer circumference of the coin to be retained, which means we can't drill the hole right at the edge. We need some minimum wall thickness, otherwise we don't have a coin any more, we have more of a crescent moon.

Limit cases

The limit cases are:

1.) 0-sized hole: we can trivially see that this does not affect the mass at all, the centre of which remains in the centre.

2.) Maximum-sized hole: this leaves us with a symmetrical ring, with the centre of mass also in the centre.

Hole locations

We can also see that for any size of hole, if the hole does not touch the perimeter (modulo the minimum wall thickness), then we can produce a corresponding coin with the same-sized hole, where the hole does touch the perimeter, and this second coin will have a more eccentric centre of mass than the first.

Maths

So for a given coin radius (r_coin) and minimum wall thickness (w), we only have one variable: the radius of the hole (r_hole). We'll work with radii instead of diameters because it makes the maths easier.

Since we're looking at a 2-dimensional cross section, the mass of the coin is proportional to the area of the shape in cross section.

So the mass of the original coin is proportional to its area, m_coin = πr_coin².

The mass removed when we drilled the hole is proportional to the hole's area, m_hole = πr_hole².

To leave a gap of w between the circumference of the hole and the circumference of the coin, y = r_coin - r_hole - w.

And to work out where the resulting centre of mass lies, we'll say the coin is a body at the origin, with a mass of m_coin, and we'll say the hole is a second body with (negative) mass -m_hole, positioned with its centre at a distance of y away from the centre of the coin.

Weighting the mass of the hole by its location, the radial position of the centre of mass is therefore r_mass = m_hole * y / (m_coin - m_hole)

So r_mass = πr_hole² * (r_coin - r_hole - w) / (πr_coin² - πr_hole²) = r_hole² * (r_coin - r_hole - w) / (r_coin² - r_hole²).

To work out where r_mass is maximised, we can find the partial derivative of this function with respect to r_hole and solve r_mass' = 0.

...at least, that was the plan, but differentiating that function proved too inconvenient.

So let's lose a variable: scale everything so that it's proportional to r_coin. Now we have:

R_coin = 1
R_hole = r_hole / r_coin
W = w / r_coin
r_mass = R_mass * r_coin
R_mass = R_hole² * (1 - R_hole - W) / (1 - R_hole²)

This is still too difficult for me to differentiate by hand. With variable names substituted for single letters, WolframAlpha gives:

and

Which is... not what I was hoping for. Did I make a mistake?

Charts

I gave up on trying to find a closed-form solution at this point and brute-forced it with a spreadsheet, for W=0.1:

When the minimum wall thickness is 10% of the radius of the coin, the optimum eccentricity (with centre of mass almost 20% out from the centre) is achieved when the radius of the hole is about 73% of the radius of the coin, which looks about like this:

That's a much bigger hole than I expected! That's why it's important to calculate these things rather than guessing. Looking at the shape of the chart, we also see that if there is likely to be some error in the size of the hole, it's best to err on the side of making the hole too small than too large.

W=0.1 corresponds to a 1mm wall thickness on a 20mm diameter (10mm radius) coin, which I think is pretty reasonable. But how much is that limiting us? How much better does our eccentricity get if we allow a slightly smaller wall thickness?

Centre of mass now almost 27% out from the centre, or about 35% better. And here are the two implied optimal coins, overlaid, and with centres of mass marked:

(W=0.1 in black, W=0.05 in red, "+" marks centre of original coin, dots mark centres of mass)

Conclusions

1.) Always position the hole as close to the edge as possible.

2.) For a 1mm wall on a 20mm coin, drill a ~14.5mm hole. The resulting centre of mass will be about 2mm away from the centre point of the coin.

3.) For a 0.5mm wall on a 20mm coin, drill a ~16mm hole. The resulting centre of mass will be about 2.6mm away from the centre point of the coin.

1. Docker

Protohackers is the second "important" service I've deployed with Docker, and the second time I've regretted it.

The big problem is that there's a short period of unnecessary downtime on every deployment.

The API server is a Mojolicious application running under hypnotoad. Hypnotoad supports downtime-free deployments out of the box. Just run hypnotoad foo, and the running instance of foo is gracefully upgraded to the new version on disk. Great success. Only it doesn't work if your application is running in Docker, because Docker containers are meant to be immutable. So instead of updating your program code and asking hypnotoad to restart it, you have to stop the container and start a new one, and the application is down in the meantime.

The user-facing website is a Next.js site, compiled to a static site and served from nginx. This should be easy-mode for downtime-free deployments! Just replace the site content and do nothing else and nginx will start serving the fresh content. Sadly, the website is also deployed as a Docker container (running nginx inside the container to serve the static site), so when the site needs to be updated, the container needs to be stopped and a new one needs to be started.

The solution checker is a Perl daemon that consumes jobs from a beanstalk tube and executes individual problem checker scripts, depending on the problem. Most of the time I'm changing the individual checker scripts rather than the "orchestrator", so most upgrades would be free here, if only I were updating the files in the container instead of destroying the container and making a new one. (Even if I need to update the "orchestrator" part, that should be trivial: I could start an instance of the new checker and send a signal to the old one to tell it to finish up its existing jobs, not reserve any new jobs, and exit when done - this would also work with Docker but is inconvenient to do with docker-compose).

So every significant component of my application can in principle be deployed with no downtime, and yet the website is going down for 30 seconds every time I fix a typo in the problem text. For what?

The benefits of using Docker are that it simplifies daemon supervision (compared to writing systemd units or something), and it makes sure that the versions of dependencies are the same in development as they are in production. But writing systemd units is only a 1-time effort, and not substantially more complicated than writing a docker-compose.yml. And maybe I don't care too much about keeping dependency versions in sync? So I think the plan is to move away from Docker and just run everything directly on Linux, with shell scripts to handle deployments.

2. Problem statements

Every problem statement so far has had some part that is ambiguous or misleading.

It's very difficult for me to dispassionately read the problem statement without already knowing anything about the protocol it is trying to specify, so I end up not writing down all of my assumptions. Different people have different preconceptions about how a protocol is likely to work. For example, I might assume things like:

• a request-response protocol can have multiple requests on a single session
• when lines are separated by "\n", the final line has to end in a "\n" as well
• byte order in network protocols is always big-endian
• the important abstraction of a TCP-like protocol is a byte stream, not a packet stream
• when measuring the length of a payload, you want to count the length of actual data after interpreting escape sequences

And someone else might have different preconceptions. A good problem statement makes these assumptions explicit, but it's hard to think of all the things that you are taking for granted, whilst you're taking them for granted.

I expect the thing to do is to recruit a playtester or two to read the problem statements before they're released, implement a solution, and then tell me what was ambiguous or misleading. If you want to do it, get in touch. The upside is that you get to solve the problems sooner than everyone else, and you get to contribute to making the problems better. The downsides are that you aren't allowed to talk to anyone about the problems until after they're released, and you aren't allowed on the leaderboard. And I can't pay you (yet).

3. Checker error messages

People consistently complain that the error messages from the checker are not clear enough. This is quite frustrating, as I actually do go out of my way to make the error messages helpful, it's just that there are a lot of different ways to fail the checks, and I don't have time to make them all maximally helpful. Most users only ever encounter a small subset of the possible error messages, and obviously they think that those particular messages should have more work put into them. (And probably they're right).

Quite often the problem is that the user has sent a string of text (maybe thousands of characters long) that is not correct. Just printing out both strings and saying they're different is not likely to be substantially helpful when they're longer than trivial examples. So I think I want a general-purpose string diffing function that finds the first difference in 2 strings and elucidates it succinctly. That wouldn't be massively complicated to do.

Apart from that, another common problem is that the user has sent a binary object with a field or two missing, so the checker is stuck waiting for more bytes before it decides what to do. Maybe at the point that the check times out it would be useful if it dumped some information about the contents of its buffers. This isn't super easy to do, because the timeout is handled at a different layer, but it would be possible. At least it would be useful to somehow give an indication when the checker has received a partial message and is waiting for more. It's quite tricky to think of a general-case solution here, it would have to work differently for every problem.

I think if I get a playtester or two, they're likely to have suggestions about the error messages that they encounter during testing, so that would help me work on the error messages more effectively.

Another view is that the checker shouldn't be providing useful error messages at all! The users should debug their own applications by looking at what they're actually sending and receiving and comparing it to what they expect. Having the checker provide "useful" error messages just causes the users to treat the checker as an automatic debugger, and then get frustrated when it doesn't work by magic. Maybe it would be better if the error messages were actually worse, and there was a tacit understanding that debugging the program is the user's problem rather than the checker's.

4. Frequency of problems

Making the problems is quite time-consuming. I was planning to do one problem every 2 weeks, but I've pushed the next one out to 3 weeks away, and it might turn into every 3 weeks now.

I originally picked a 2-week period because that makes 26 problems a year, and Advent of Code has 25 problems a year, so I thought it would be in the ballpark of reasonability. But the amount of time I spend on making Protohackers problems has given me a newfound appreciation for how hard Eric Wastl must work on Advent of Code.

Coming up with problem ideas hasn't been too hard, because I still haven't run out of principles that I want to cover. And writing a reference solution for each problem isn't too hard either. The hard parts are 1.) trying to make a checker that checks all of the important parts of the problem, without assuming any behaviour specific to my reference solution, and 2.) trying to write a problem statement that accurately and succinctly conveys the protocol specification, with every important behaviour explicitly documented.

5. User acquisition

There are 852 user accounts at the moment, of whom 399 have completed any problem, of whom 261 have solved any non-echo-server problem.

I don't know how most people are finding the site. I'm not advertising anywhere. It would be good to be more purposeful about attracting new users, but I don't have any great ideas. I Tweet, a little bit, but the Twitter account only has 74 followers so I don't think it does a lot of good. I emailed a handful of programming newsletters to ask if they would mention it, but none of them were interested.

I need better ideas here.

6. Monetisation

Apart from being annoyed by the ambiguous problem statements and confusing error messages, people like Protohackers, and frequently tell me so. If you've made something people like, then you've created wealth and should try to keep some of it for yourself in the form of cash. This would also allow me to pay any would-be playtesters without feeling like I was throwing money in a hole.

So I'd love to make money out of Protohackers, but I'm not sure how viable it is until there are a lot more users. Ideas include:

Ask the users to pay

A couple of people have already offered to make a donation, but I don't have a good way to accept ad hoc donations, and I don't want people to feel obliged to donate. Maybe I could set up a Patreon, but I can't see more than 5% of people wanting to pay anything, and 5% of 260 arguably-active users is, like, 13 people? Paying a few quid a month? Doesn't seem worth it.

Sell merchandise

I'm very picky about clothing quality, and can't recall ever buying branded merchandise that I was actually satisfied with, so I'd probably spend a lot of time and money trying out different suppliers. It also has the same problem as "asking users to pay" where I'll earn a tiny amount of money each from a tiny handful of people, so it's not really worth doing.

Solicit recruitment ads

I think the best option is recruitment ads. The Protohackers userbase is highly targeted towards competent programmers, and in particular success at Protohackers implies competency at network programming, and I imagine hiring competent network programmers is incredibly valuable to some tech companies. Much more valuable than anything else I can imagine anyone wanting to advertise on Protohackers.

So putting recruitment ads on the site is probably the best idea. Then the problem becomes who exactly wants to recruit network programmers? How do I contact them? How much do they want to pay?

So if you want to advertise on Protohackers, or you know someone who might, please get in touch. For the time being, you can simply thumb down the leaderboard, click on the GitHub links, and contact people yourself without paying me anything. You would only want to pay if you want to invest in a recruitment pipeline, rather than a 1-time mailshot to a handful of candidates.

Here's a rough sketch of the stack that implements human minds:

We are approaching the problem from two different directions. We can study the physical properties of the brain (working upwards), and we can introspect on thoughts (working downwards). You'd think this would allow us to meet in the middle and understand the system all the way to the bottom.

Sadly not. The cloudy "???" section may have unimaginable complexity that is neither physically a part of the brain, nor accessible to introspection. Just imagine trying to understand how a web browser works if you understand nothing beyond electronic logic gates, except that you get to point and click in the GUI.

You would have no idea about the unimaginable tower of software complexity sitting underneath Firefox, and no studying of logic gates will explain it. Firefox seems to behave like a completely different type of thing to logic gates. And it is a completely different type of thing. You need to understand an enormous stack of abstractions before you can comprehend Firefox in terms of logic gates.

I think trying to understand the mind in terms of neuroscience similarly disregards the tower of complexity that lies in the cloudy "???" zone.

Furthermore, the transistors and the logic gates are not necessary for Firefox to function. They're just a substrate on which we can perform computation. Any substrate would do! Similarly, I think brains and neurons are not necessary for minds to function: they're just a substrate which can interface between biology and whatever magical yet-to-be-discovered abstractions lie in the "???" layer.

And you may think that understanding those abstractions must be very complicated. Well, the way birds work is very complicated, but the principles of flight are not, and flight only seems complicated if you try to understand it by trying to understand birds.

So if not neuroscience, then who will help us with the hard problem of consciousness? I think the problem lies at the intersection of computer science and philosophy. The "mind" part is very obviously the domain of philosophy. I think that if you work your way upwards from "neuroscience", you eventually derive a biological implementation of computer science, and then you're stuck again.

Why do I think that? I claim that a brain is purely an information-processing machine. (If you disagree, your counter-argument should probably start by outlining some proposed input or output of the brain that it would not be possible to encode numerically).

We know from the Church-Turing thesis that general-purpose computers can compute all computable functions. Since the brain is (I claim) just computing a function, all of the operations of a brain can be performed by general-purpose computers. And since (I claim) the mind is implemented on the brain, it therefore must be possible to implement a mind in software. I doubted this before but I am a believer now.

So let's skip all the hard work of explaining how neuroscience creates computer science (a worthy and interesting field of study, no doubt, but not the one I'm interested in right now). Let's assume that it is true that working upwards from neuroscience eventually reaches computer science and, given that we already have a way to implement computer science, reduce the problem to this:

Much more tractable!

Now assume we have developed a candidate "software mind", that might have its own conscious internal experience. How would we find out? How would we convince others? We can't just hook it up to a text chat and ask it, because it might say it's conscious even if it is not. We need to be able to show that it is truly conscious.

So before we start building minds in software, we ought to agree on a way to classify them. Traditionally this is very difficult, because the only way to decide whether a person has consciousness is to experience it for yourself. Since you can only access your own internal experience, the only person whose consciousness you can be sure of is yourself.

Currently our best heuristic for determining whether something else is conscious is to check how much it behaves like a human. Humans: as conscious as it gets. Apes: very conscious. Dogs: quite conscious. Plants: vegetative. Rocks: nothing going on, nobody home, not at all conscious.

This works surprisingly well in ordinary daily life, but is not very effective at identifying a mind that isn't like a human. It would be like checking whether a plane can fly by measuring how much it looks like a bird.

If we are creating minds in software, however, we have the advantage that we have total access to all of the internal state of the system, while it's running. So what things would we look for in a hypothetical software mind to determine whether or not it's conscious?

I don't know. But maybe answering that would give us some pointers on how to construct one.

I suspect the kind of people who might be good at this puzzle are those who do a lot of CAD and those who do a lot of lathe work, because they both involve grappling with the equivalence between the 2d profile and the 3d revolved shape.

Profiles

Here's a rough profile of a chess pawn, and the shape you get by revolving it around the tall vertical edge:

The idea behind "alternative revolutions" is that we can revolve the same profile around a different edge to get a different object:

Hopefully you see that this has the same profile, but is a completely different shape.

We also have the option to make negatives by taking a negative of the profile:

This is the same profile, except now the right hand side, instead of the left hand side, is "inside" the shape.

And we can do the same with the horizontal axis instead of the vertical axis. For some reason the 3D shape created by this profile is very confusing to look at, but it should make sense if you keep remembering the profile that it's made out of.

(This part would exactly mate with the first "alternative revolution" above).

And the same thing with the other horizontal edge:

The mating part for which is:

We can sweep around an axis that is spaced away from our profile, to leave a hole in the middle:

And our profile can either face outwards (above) or inwards:

And the same with the horizontal rotation axis:

And of course the same is also possible with the negative version of the profile, which among other things lets us make a mating part for the standard chess pawn:

We also have the option to tilt the rotation axis if we want to:

This lets us smoothly interpolate between the version with the vertical rotation axis and the one with the horizontal rotation axis:

With a bit more creative license, we could decide that we don't want the hollow cones that give the game away, and we could easily cap off the top and bottom with flat surfaces.

(Sadly I can't think of a way to smoothly interpolate between the positive and negative version of the profile.)

So there are quite a lot of different shapes we can make, given just one profile, by having the option to turn the profile negative, and varying the position and direction of the rotation axis.

Puzzle ideas

Identify common objects: We give the player a bunch of objects that are made by revolving common object profiles (some positive, some negative) in uncommon ways. The player has to figure out what common object each shape is based on, and then... what? Concatenate the words to form a password? Just gain the satisfaction from having solved it?

Matching pairs: We give the player a bunch of objects that are made out of contrived profiles (i.e. arbitrary shapes we make up, not necessarily related to objects that the player would know). The player has to work out which of the objects share profiles and pair them up. And then... there's a number on some of the objects and a letter on the others, and they have to put the letters together in order of the numbers to make a password?

Odd one out: We give the player a bunch of objects that can be made out of contrived profiles or common ones, and they have to locate the only object that doesn't share a profile with any of the other objects (or, along similar lines, they have to locate the only pair of objects that do share a profile).

Data encoding: We can come up with a way to encode a small amount of data in the profile (as an existence proof: make a large radius for a 1 bit and a small radius for a 0 bit, but more interesting stuff is possible) and the player gets the objects and has to decode the data we've given them. Maybe in combination with "odd one out" or "matching pairs" to work out which pieces of data are actually useful and in what order?

Normally I prefer (dis-)assembly puzzles, or something that acts like a box and you have to solve the puzzle to retrieve an object, but I can't see a way to readily adapt the "alternative revolutions" to anything interesting. Probably the best option is to just use it as a matching puzzle to encode a combination for a padlock in an escape room.

What better ideas do you have?

Fly.io can host anything that can run in a Docker container, and you don't even need to write your own Dockerfile because it automatically handles the most common languages and frameworks. I wish the tooling were less verbose, it's as if it's got debug mode logging permanently enabled. It is also quite slow. In principle we don't need any more than building a tiny binary and scping it to a server, why does it take multiple minutes to make a deployment?

Using Fly.io is slightly more trouble than using a normal VPS, so if you already have a VPS you will probably find that easier. Depending on your ISP, your router, and how comfortable you are with port forwarding, Fly.io could be either more or less trouble than hosting your program at home. If you think hosting at home sounds easy, then that is probably easier for you than Fly.io.

Let's assume that you don't want to pay for a VPS, and you can't host at home (maybe you are behind CGNAT). In that case Fly.io is your next best option!

Firstly, install Fly.io using the instructions on their website, which is currently:

$ curl -L https://fly.io/install.sh | sh

Follow its instructions to put its environment variables in your bashrc. Signup for Fly.io:

$ fly auth signup

This will try to open a page in your browser to let you either connect a GitHub account or sign up with email.

Make a directory for your project and copy the Go example program from the bottom of the Protohackers help page into main.go.

Fly.io is supposed to be able to auto-detect a Go program and deploy it with just fly launch, but I have found it doesn't auto-detect Go unless you have a go.mod. (You may have better luck with other languages). To help it detect your Go program, run:

$ go mod init example.com/foo
go: creating new go.mod: module example.com/foo

Now we'll launch it on Fly.io. I accepted the defaults for everything.

$ fly launch
Creating app in /home/jes/protofly
Scanning source code
Detected a Go app
Using the following build configuration:
	Builder: paketobuildpacks/builder:base
	Buildpacks: gcr.io/paketo-buildpacks/go
? App Name (leave blank to use an auto-generated name): 
Automatically selected personal organization: James Stanley
? Select region: lhr (London, United Kingdom)
Created app old-surf-8338 in organization personal
Wrote config file fly.toml
? Would you like to set up a Postgresql database now? No
? Would you like to deploy now? No
Your app is ready. Deploy with `flyctl deploy`

The default Fly.io configuration is to run an HTTP proxy in front of your application, but that's not what we want.

We need to edit the generated fly.toml file to make the [[services]] section expose TCP directly rather than proxying with HTTP. There is more information in their UDP and TCP documentation.

Clear out the [[services]] section of fly.toml and replace it with:

[[services]]
  internal_port = 10000
  protocol = "tcp"
  
  [[services.ports]]
    port = 10000

And deploy the application with fly deploy. This part is extremely verbose. It would benefit from the Rule of Silence. It also takes quite a long time. You just have to wait.

$ fly deploy
 
[... hundreds of lines redacted ...]
 
 1 desired, 1 placed, 1 healthy, 0 unhealthy
--> v2 deployed successfully

Once our application is "deployed successfully", it's up and listening on the public Internet. Despite the hundreds of lines of output that we were shown, I think we actually haven't been told what IP address our program is listening on. In my case the application was called old-surf-8338, and we can find out what IP address it's listening on by resolving old-surf-8338.fly.dev:

$ host old-surf-8338.fly.dev
old-surf-8338.fly.dev has address 168.220.85.87
old-surf-8338.fly.dev has IPv6 address 2a09:8280:1::a:77b4

So in my case we'll put 168.220.85.87 and port 10000 into the checker form:

And the check should pass!

Great success indeed.

This is something I've been planning to make for years, but kept forgetting about. I would only remember it when in the passenger seat of a car on the motorway at night time, fascinated by the patterns of moving lights.

Here's a small embedded version, if your browser runs JavaScript:

For the full version (with music!) click here: Nightdrive.

Gina said it's a great way to find out how filthy your monitor is.

The entire scene is created purely by drawing circles on a HTML5 canvas. It actually works a lot better than I expected. The biggest flaw is that the cars are totally transparent, so you can see lights from distant cars even where they should be occluded by closer cars.

Motorway simulation

The core simulation is entirely 2-dimensional. The coordinate space looks like this:

The motorway is always parallel to the y-axis.

Each car has a 2d position vector and a y-axis velocity (cars on our side having positive velocity, and cars in the oncoming carriageway having negative velocity).

400 cars are positioned at 25 metre intervals, in random lanes, at initialisation time. They also have some slight variation in the size, position, and colour of the lights. Cars in the left-most lane travel at a constant simulated speed of about 40mph, in the middle lane at 70mph, and in the fast lane at 100mph.

When any car is "behind" the viewer (lower y coordinate), and with a lower velocity (will get further behind) it is wrapped forwards by 10km. Similarly, when any car is 10km "in front of" the viewer (higher y coordinate), and with a higher velocity (will get further ahead), it is wrapped backwards by 10km.

That's enough to give the illusion of an infinite road with infinitely-many cars, travelling in both directions, and at different speeds in different lanes.

Projection

A top-down view of a 2d world isn't the effect we're going for, so we need to first make our world coordinates 3-dimensional, and then project them on to the 2-dimensional screen.

To make the coordinates 3-dimensional, I just add a "height" to every point in the 2d simulation. This works because all of the elements I need are at constant heights above the ground: car headlights, cat's eyes, and street lights.

My projection into screen space is not necessarily mathematically sound. I just fiddled with it until it looked how I wanted.

To turn a point from world coordinates into screen coordinates we first subtract the observer's position, to get a position (x,y,z) relative to the observer.

If the relative position vector has a negative y coordinate, then we know the object is behind the viewer and therefore not visible, so we don't draw anything.

Otherwise, we find the distance to the object by taking the magnitude of the relative position vector:

dist = sqrt(x² + y² + z²)

Then our coordinates in screen space are:

x_screen = x_centre + k * x / dist
y_screen = y_centre - k * z / dist

Where k is a scale factor that sets the size on the screen, and (x_centre, y_centre) is the centre of the screen.

(We need to subtract for y_screen because in screen space positive y is down, but in our world space positive z is up).

I found it actually worked better if I omitted the x offset from the distance calculation (so just sqrt(y² + z²)), otherwise objects near the edge of the field of view were weirdly distorted. But I wouldn't suggest doing that in the general case.

You can look at the actual projection function I used if you like.

Terrain

To make the road a bit more interesting we can add some hills. I wrote a function adding up some arbitrary sine waves to make something that seemed reasonable:

function terrain(y) {
    return 10*Math.sin(y/1000) + 5*Math.cos(y/527) + 2*Math.sin(y/219);
}

This takes a y coordinate in road space, and returns the height of the road at that point. This is then added to all z coordinates by the projection function.

Looking at the road from the side, the terrain it generates looks something like this:

(x on the graph is y in world-space, y on the graph is z in world-space, both in metres).

Road occlusion

At this point we have quite a glaring issue: we can see through the ground!

Everything the other side of the crest of the hill should be occluded by the hill, but since all we're doing is projecting positions of lights, nothing can ever get occluded by anything.

A point P is occluded by the road if there is any position, between the viewer and P, that has a screen-projected road height "higher" (smaller value) than the screen-projected y coordinate of P. Because that means if we were to actually draw the road, it would get drawn over the top of P.

I sorted all rendered lights by distance from the viewer (closest first), and walked along the list, keeping track of the highest screen-space road height so far encountered, and occluding any points that fall "below" that height. That means we'll occlude any points that fall behind the road position at any closer position that has a light on it. This is good enough, because we always have frequent-enough sampling of the terrain function due to the density of cars and street lights.

You can look at the implementation if you like.

Come to think of it, I expect there might be an analytical solution to determining whether any given point should be occluded by the road. Maybe that would be better! We just need to find out whether the straight line passing through the camera and P intersects the road at any point between the camera and P: if it does then P is occluded by the road.

Music

The background music is from this video which describes it as "Royalty Free".

I've been told that the audio quality on Nightdrive is bad. I did this on purpose to reduce the size of the mp3, and it doesn't sound bad to me. If you think it sounds bad then you should mute Nightdrive and play that YouTube video in a different tab.

More

There are a few more things that I think would be fun to do:

Bends: Instead of having a perfectly-straight road, it would be good to add some bends, so the patterns of lights would curve off into the distance, something like this:

(From jonnywalker on flickr)

Street light occlusion: Street lights aren't just magical floating orbs, they're held up by metal poles. When something passes behind the metal pole, it is momentarily invisible. This would not be too hard to do in Nightdrive. We'd just use a similar process to the "road occlusion" to hide lights that are hidden by the poles.

Car occlusion: Cars also are not just magical floating orbs, they have volume and are opaque. This would be slightly harder than street light occlusion. Probably a first pass would be to render a black cuboid behind each car's lights, so that the cuboid blocks out anything that would be blocked by the car. This is inconvenient because currently we can only render circles.

Speed variation: Not every car in the same lane drives at exactly the same speed. Sometimes you catch up to the car in front, or the car behind catches up to you. Adding speed variation is easy, but comes with the problem that cars end up driving through each other, which for some reason feels unrealistic. This brings us on to...

Lane changes: When a car catches up to the car in front, it should indicate right and move over to overtake. When it has passed and there's a space on the left it should indicate left and move back over. We could give each car a sort of "personality" which tells it how close it can get to other cars, how long it should indicate for before changing lanes, etc.

Rain shader: A rain effect might be too much trouble to do with just a canvas. Maybe it would need WebGL. But I would like to apply some sort of rain shader to the rendered image to give the effect of driving at night time. We can imagine having a collection of "droplets" on the "windscreen" which refract the light passing through them (dots that distort the pixels underneath). Periodically we could sweep a windscreen wiper across the field of view to clear the dots, and we can randomly add dots at some rate based on how heavily it's raining.

Game: Finally, I wonder how we could turn this into some sort of game? I'm especially not looking to make a driving game. I still want the car to basically drive itself. What game can we make where the premise is that you're a passenger on the motorway at night time? It shouldn't be a particularly taxing game, I think the main experience should still be that you're just enjoying watching the lights, but it would be cool if there was some interactivity and some sort of goal.

So far, 499 people have signed up for the site (+437 since last time). 199 have solved the test problem (+170) and 76 have solved the latest problem (+62).

The reason behind the big increase in users is that there was a thread about Protohackers on the HN front page for a little while.

The problem statement

So far there has only been one point of confusion highlighted in the problem statement (a big improvement over last time!).

The problem statement said:

Because each client's data represents a different asset, each client can only query the data supplied by itself.

To my mind it was obvious that each connection was a separate client, but some people interpreted it to mean that only unique client IP addresses are separate clients, and that every connection from the same IP address counts as the same client.

I've updated the problem text to say:

Each connection from a client is a separate session. Each session's data represents a different asset, so each session can only query the data supplied by itself.

Ideally the problem statement would never get changed for any reason, so that every user always sees the same text. However, where the problem statement is confusing, I consider that a bug, and bugs should be fixed.

Someone pointed out in the Discord chat that it is very weird for clients to be "analysing historical price data", but permanently lose access to it as soon as they disconnect. It is weird, I agree. The intention for this problem was that you need to persist state across multiple requests (compared to the previous problem where you didn't have to persist any state), but you do not have to share state between multiple clients.

Email deliverability

A handful of people have not been receiving the signup/login emails. Two American universities are bouncing the mails with a message about poor domain reputation. I have emailed postmaster@ their domains to ask to get unblocked.

Two (would-be) users have email on their own domains hosted by Fastmail, which seems to be silently blackholing the mails with no bounce message. I've opened a support ticket with Fastmail.

So I don't know if I will have to add some alternative non-email-based login flow, or whether the problem will get solved on its own over time as a.) I complain to people who are blocking the emails, and b.) the domain reputation improves as it ages.

Upcoming features

Overall leaderboard

I still haven't done the overall leaderboard. The current plan is that if N people have ever successfully solved any problem, then your rank for a problem you haven't solved yet will be N+1, and the overall leaderboard will be ranked by your summed rank, ascending.

(So if there are 3 problems, and you got 1st place on all 3, your score is 3. If you got 2nd place on all 3, your score is 6, which is higher than 3, which is a worse overall ranking).

I don't like the fact that in cases where people have not solved all problems, their leaderboard positions can be changed even if they don't do anything (just because new people solve some problems, which increases N), but I think it is better than any alternative I can think of, and in particular it is stable for those users that have solved all problems, and it means you always get an improvement in your overall score by solving a new problem.

I am keen not to create any overall leaderboard until the logic is definitely settled, because I don't want to give someone 1st place on the leaderboard and then change my mind and hand it to someone else!

IPv6

Several people have asked about IPv6, including one person who has a VPS that apparently only has IPv6 connectivity. So I plan to look at adding IPv6 support. I am wary that this will double the surface area of possible connectivity issues, but I think it is worth doing.

The most important part is that the checker can access people's servers using IPv6, but it would also be nice if the web site was available over IPv6.

Dashboard

I have an InkyWHAT screen hanging from a shelf next to my desk with a 3d-printed bracket:

It shows the total number of users, problem attempts, and problem solutions, as well as information about the most recent accepted solution, the current time, and the server's CPU load.

Mainly it just saves me from having to manually look this stuff up in the database. It gets its data from a special (authenticated) API endpoint.

If you have an InkyWHAT and you don't like how it totally fades to black every time you update the screen, you should use the InkyFast wrapper from pwnagotchi - it uses an alternative lookup table for the display driver so that it updates more quickly. It can leave a tiny amount of ghosting, but it's a big improvement over the whole display fading to black every 10 seconds.

Mine looks like this:

It has a piece of diamond at the tip.

You don't turn the spindle motor on, you're just using the spindle to hold the tool. Being spring-loaded means it conforms to slightly out-of-level surfaces while maintaining almost constant pressure.

Ring

Yesterday I used the drag engraver with my weird rotary axis to engrave a commemorative ring I made for an upcoming special occasion. The quality of the engraving was much better than I expected, so I want to do more of it.

I want to make a ring that has a huge amount of tiny text engraved on it, but first I need to know how small I can make the text before it becomes illegible.

Test card

I drew up a little test card in Inkscape:

This has text from 4mm down to 0.5mm in 0.5mm steps, and then down to 0.2mm in 0.1mm steps.

(Incidentally, I find Inkscape very annoying to use for engraving toolpaths, because Inkscape includes the thickness of a line in its dimensions. This means when you resize a piece of text to be 4mm tall, you actually end up slightly under-size, because the thickness of the line is included in the "4mm" measurement).

This text is generated with the "Hershey Text" plugin using the "Sans 1-stroke" font.

I engraved it on an offcut of 1mm aluminium sheet with the CNC machine:

(It turns out it's quite hard to take a picture of a very shiny piece of metal - the text is much easier to read in-person).

The aluminium sheet is a much softer material than the ring. I don't actually know what metal the ring is made of, it's some surprisingly-hard type of brass or bronze. It was a small piece of scrap left in my garage by the previous owner of the house.

Due to the softness of the aluminium, the engraved lines on my test piece are quite a bit deeper than on the ring, which tends to make the smaller text harder to read because adjacent lines blend into each other.

Conclusion

All sizes from 1mm and up are legible.

I reckon 0.5mm is legible if you squint. 0.4mm and 0.3mm are legible without magnification only if you already know what the text says. I don't consider 0.2mm legible even though I know what it says.

Given that the lines we'd get on the ring will be thinner, I think it is safe to go below 1mm. Possibly I'd suggest 0.6mm just to be on the safe side.

Under the microscope

Just for fun I took some pictures with the USB microscope:

0.2mm:

0.3mm:

0.4mm and 0.5mm:

0.4mm and 0.5mm again:

Drag-engraved PCBs

I have made a circuit board with isolation routing before, but I found it very difficult. In total I have had maybe 7 attempts at making PCBs with isolation routing, with only the 1 success.

The major problems I had were:

• the surface is never truly flat, which means the cut depth is different in different places; it might cut so deep that it obliterates your trace in one area, while it's not even isolating the traces in another
• the width of the cut is too big because the tiny v-bit tools are rarely concentric enough

You can solve the flatness problem by probing the surface and correcting for its shape, but I haven't established a good workflow for that. I'm told that bCNC is the tool to use, but haven't got around to trying it yet.

I have quite a lot of fine-tipped v-engraving bits, but I have broken the one I used to create the successful PCB. All of the others, despite having very thin and precise tips, are not concentric, which means the tip spins around in a big circle and routes a slot much bigger than the size of the tool. Buying more expensive tools would probably help, but I break them so frequently that I'm not prepared to pour more money into them.

Anyway, my big idea here is to try using the drag engraver to do the isolation routing! It kills both birds with one stone: it automatically corrects for an out-of-level surface because it's spring-loaded, and it gives you an extremely thin cut because it's not spinning.

As long as the drag engraver can provide enough pressure to cut all the way through the copper I don't see what could go wrong. So I plan to try this tomorrow.

GPT-3 takes a prompt and creates more text. Stable Diffusion takes a prompt and creates an image. Copilot takes a prompt and creates source code.

We only need to imagine a very small number of new technologies, along similar lines, in order to create 3-dimensional worlds, and interaction logic, from simple text prompts.

Just imagine an extra layer on top of Stable Diffusion that takes the same kinds of prompts, but it generates you a 3d world instead of a 2d image.

Imagine an extra layer on top of Copilot that generates an entire application rather than just small functions, and prime it to create common video game interaction patterns rather than leetcode solutions.

Maybe it would require very detailed prompts? OK, let's imagine using something like GPT-3 to turn a single coarse-grained "world prompt" into the fine-grained prompts for the underlying layers.

Now put on your VR headset. Connect to your friend who lives on the other side of the world. Give it a world prompt:

I'm a cop, my friend is a robber. I am chasing him through the streets of Chicago at night time. In the rain. If I catch him within 15 minutes I win, otherwise he wins. Also we each have a radar on our wrist that tells us the direction to the other player. Go!

The VR system comes with a built-in concept of what a 3d world is, how your motions in the real world control a character in the virtual world, how to link up multiple players into a single world over the Internet, etc.

Copilot++ has created the tiny bit of interaction logic to set a 15-minute timer, work out whether you've "caught" the robber, and implemented the standard "radar arrow on the wrist" pattern that is probably just a Unity++ plugin.

StableDiffusion++ has created the 3d world (both geometry and textures) and continuously paints in new areas of the world as they come into view.

So now you've got a reasonably photorealistic, fully-interactive, immersive 3d world, created to your specification, and a game to play against your friend, also created to your specification. You're almost actually a cop chasing a robber through the streets of Chicago at night time. In the rain.

And you can set the prompt to whatever you want! Literally anything you want. Literally everything is possible. It's like AI Dungeon on steroids.

When you ask it for a world, your experience would be exactly like doing the thing for real, except maybe sometimes the people have extra limbs and none of the text makes sense.

Why can't this exist? I think this can exist. I think the only question is how long it's going to take.

So far, 62 people have signed up for the site (much more than I expected!), of whom 29 have successfully solved the test problem. 19 have attempted the first "real" problem, with 14 having solved it successfully. It's not clear whether the other 5 have been discouraged and walked away for good, or whether they'll be back for another go.

Amusingly, first and second place on the leaderboard were separated by only 2 seconds!

I don't know if these two were communicating or not, but they were submitting different IP addresses, so I assume they were not working together.

Discord server

Two people emailed me and asked about setting up a Discord server for people to join, so now there is one. If you want to join, you can click: https://discord.gg/RqqmGePnWU.

The problem statement

There were a few issues with the problem statement that I hope not to repeat in future problems.

The initial release of the problem statement didn't explicitly allow clients to make multiple requests over one connection! That's bad, because it means depending on your preconceptions of how the protocol is likely to work, you might assume that the client makes a single request, receives a single response, and then disconnects. Multiple people implemented servers that assumed only a single request per session was allowed. I have clarified this now.

The problem statement initially stated that the "number" field of the request object needed to contain a number in order to be considered well-formed, but some people assumed this meant it needed to contain an integer. While I maintain that a thorough and pedantic reading of the problem statement would lead to the correct interpretation, that shouldn't be the standard: a casual reading by a competent programmer should be enough to arrive at a correct understanding, especially when you consider that many users may not speak English as a first language.

Eric Wastl (creator of Advent of Code) says something like:

for each sentence in the problem statement, there is at least one user who reads everything except that sentence.

The intention was that non-integers are still accepted as well-formed requests, but that they are not prime (because only integers can be prime). But this is really just a distraction from the core purpose of Protohackers, which is to write programs that implement network protocols. Annonyingly-pedantic input validation is not the goal. I've updated the problem statement to be more explicit about the intent, and will try to avoid doing this sort of thing again.

One person was confused about exactly what I meant by a "newline character":

Each request is a single JSON object followed by a newline character ('\n', or ASCII 10)

They were outputting the literal string \n (i.e. a backslash followed by an "n"). I'm not really sure what to do about that. Maybe I should hyperlink "newline character" to the Newline Wikipedia page?

The checker

A weird quirk of the protocol is that the correct behaviour, in response to a malformed request, is to send a malformed response, rather than some sort of standardised error response. I just thought this had an amusing symmetry.

Unfortunately, my checker was not correctly interpreting all malformed responses. It only considered invalid JSON to be a malformed response, even though the problem statement said that a response was "malformed" even if it was valid JSON, but had fields missing, or invalid values in the fields. I had failed at making a sufficiently-pedantic interpretation of the problem statement in exactly the same way I was trying to test the users' interpretations of the problem statement. An amusing symmetry indeed. Hoist by my own petard! (This is fixed now).

One of the test cases is a really long number (something like 63 digits long). These are selected at random and almost always have low factors, so it shouldn't be too much of a problem to test them for primality. But sometimes bigints are too inconvenient. What then? At least one person noticed that the only test case that had a number larger than 64 bits also had a spurious extra field in the JSON ("bignumber":true), and he simply tested for this field, and returned a "non-prime" response without even checking the number! Lol.

Logo

I paid a man on Fiverr £15 to design a logo for Protohackers. The brief was something like "I want a logo that will fit with the existing colour scheme, and convey programming and networking". He came up with these three:

I can kind of see what he's trying to do with angle brackets on the first one, although it seems a bit more "HTML" than "HTTP". The second one clearly conveys the Internet, and connectivity, but I can't help seeing it as a cartoon snail. I'm going with the third option, but resized and rearranged to look a bit more subtle on the page:

Still to solve

I haven't yet worked out whether/how to create an "overall" leaderboard that combines the scores from the individual problem leaderboards. The Advent of Code solution is to cap the daily leaderboard at 100 places, assign 100 points for 1st place, 99 for 2nd place, etc. down to 1 point for 100th place, and add up all the points to form the overall leaderboard.

I'd like to find a solution that includes more than the top 100 places (admittedly, a leaderboard with more than 100 names on it is still a fair way off!). One option is to add up the leaderboard position for each problem's leaderboard, and then the lowest sum is the highest ranked on the overall leaderboard, but this has the drawback that people who haven't completed all of the problems can't be ranked on the overall leaderboard.

So I'm looking for a ranking solution with the following properties:

• completeness: rank everybody who has ever completed any problem
• fairness: if Alice and Bob have completed the same subset of problems, and Alice beat Bob on every problem, then she should be ahead of Bob overall
• stability: if Alice is ahead of Bob overall, and then Charlie comes along and submits a solution to a problem that both Alice and Bob have already solved, Alice and Bob's rankings are undisturbed

The Advent of Code global leaderboard satisfies fairness and stability, but not completeness.

Advent of Code also has private leaderboards, where you score points based not on 1..100, but on 1..N where N is the number of people in the private leaderboard. This satisfies completeness and fairness, but not stability, because adding a new person to the group can change the relative positions of people who were already in the group! (Because every submitted problem is now worth 1 more point: someone who got a high score on 1 problem only gets 1 more point, but someone who got low scores on M problems gets M more points).

It may be the case that there is not any possible ranking algorithm that satisfies all 3 criteria. I'm pretty sure this is not a new problem, so I probably just need to do some reading.

Currently only the "Smoke Test" problem is available. This problem is intended to make sure you have your networking setup sorted out, so that you can deploy a program that accepts connections and receives and sends data.

If you are having trouble getting the Smoke Test to pass please ask for help! I want to see you succeed. I expect the most likely source of frustration is firewall configuration, but if you are just not familiar with your favourite language's networking libraries, I'll lend a hand with that too.

The next problem will be released on Thursday at 17:00 UTC (18:00 in the UK) and my plan from then is to release a new one every ~2 weeks, but this could change based on user feedback.

Please take a look, tell your friends, and have a go at the smoke test.

Most importantly: if you think this is your kind of thing, tell me what you think about it, and what you would like to see changed. The project is currently in a larval stage, and feedback given at this time will have an outsized impact on the direction of future growth.

The link again is: https://protohackers.com/.

I don't know why it didn't occur to me to write a SLANG interpreter sooner. Possibly I thought it would be too much trouble, and only after I wrote the Lisp interpreter did I realise that it would not. There's also the weirdness that SLANG code does not have automatic memory management, and I don't know if I've ever seen an interpreted language that requires manual memory management (maybe FORTH, if you squint?). SLANG just doesn't seem a natural fit for being interpreted.

But it turns out that if you don't have to bother with a garbage collector, and you don't care if the computer crashes when the stack overflows, then writing the interpreter is quite straightforward. And these arguably aren't even defects of the interpreter, because the compiled language works exactly the same way. It's up to the programmer to make sure the program is not going to leak memory or run out of stack space.

The source code is currently slightly longer than that of the compiler:

$ wc -l slangc.sl slangi.sl
 1230 slangc.sl
 1293 slangi.sl

But this is only because I have reimplemented some of the more important parts in assembly language. Originally it worked out slightly shorter than the compiler.

If you like, you can watch a video of me using the interpreter to solve Advent of Code 2018 day 4.

Usage

The interpreter is available in the online emulator, just run your program with slangi foo.sl, for example:

$ cat foo.sl
printf("Hello, world!\n", 0);
$ slangi foo.sl
Hello, world!

It overrides the cmdargs() system call so that arguments passed on the command line (e.g. slangi foo.sl arg1 arg2 arg3) come out the same way they would do in the equivalent compiled program.

AST representation

Each node of the AST is a pointer to an object whose first element is a pointer to the function that evaluates the object. For example, you could create a node that returns a constant value like:

var EvalConstNode = func(node) {
    return node[1];
};
 
var ConstNode = func(val) {
    var node = malloc(2);
    node[0] = EvalConstNode;
    node[1] = val;
    return node;
};

The objects are then evaluated with a helper function called eval:

var eval = func(node) {
    var f = node[0];
    return f(node);
};

Hopefully you can see that eval(ConstNode(42)) will return 42.

We can easily make a node for if (cond) thenblock else elseblock that tests its condition, evaluates its "then" block if the condition is true, and evaluates its "else" block (if any) otherwise:

var EvalConditionalNode = func(node) {
    var cond = node[1];
    var thenblock = node[2];
    var elseblock = node[3];
 
    if (eval(cond))
        eval(thenblock)
    else if (elseblock)
        eval(elseblock);
};
 
var ConditionalNode = func(cond, thenblock, elseblock) {
    var node = malloc(4);
    node[0] = EvalConditionalNode;
    node[1] = cond;
    node[2] = thenblock;
    node[3] = elseblock;
    return node;
};

Composition of these different node objects gives us an object tree that represents the program structure, and calling eval() on the root node results in evaluation of the entire program.

Calling convention

Calling convention on SCAMP is that the function arguments are pushed onto the stack and the return address is passed in the r254 pseudoregister (i.e. 0xfffe).

(Why isn't the return address also passed on the stack? Because that can't be done in 8 cycles, which is the upper limit for a single instruction. In practice return addresses are often "manually" stashed on the stack, to allow nested function calls to work).

A "function pointer" in SLANG is just a number like any other. (In fact, SLANG doesn't really have types at all: every value is a 16-bit word). This means that when our interpreted program calls a function, all we get to know is what number it is trying to call.

This presents 2 problems:

1. When our interpreted code makes a function call, we need to obey the calling convention in case it is calling out to a compiled function (e.g. in library code).
2. Sometimes library code can call back into our interpreted code, so our interpreted functions need to be callable with ordinary calling convention.

I made the FunctionDeclaration parser create a machine-code shim for the function. The shim calls a function evaluator, passing it a pointer to the arguments, a pointer to the list of argument names, and a pointer to the AST for the function body. The function evaluator sets up the new local scope and uses eval() to evaluate the body.

This way each function declared in the interpreted code still has its own entry point, and can still be called with ordinary SCAMP calling convention, which means compiled functions and interpreted functions can interact freely.

break, continue, return

break, continue, and return are challenging, in that they can require jumping a long way up the interpreter's call stack. It would be most inconvenient if every single evaluator had to have special code to notice when an eval() call is asking for a break, continue, or return, and manually pass the condition upwards.

My solution was to use longjmp. (I like to say that longjmp is my goto).

The evaluator for a LoopNode uses setjmp to set globals for BREAK_jmpbuf and CONTINUE_jmpbuf. The evaluator for a BreakNode simply longjmps to the BREAK_jmpbuf:

BreakNode = func() {
    return [EvalBreakNode];
};
EvalBreakNode = func(n) {
    longjmp(BREAK_jmpbuf, 1);
};
 
EvalLoopNode = func(n) {
    ...
    setjmp(CONTINUE_jmpbuf);
    if (!setjmp(BREAK_jmpbuf)) {
        while (eval(cond))
            if (body) eval(body);
    };
    ...
};

There is a similar setup in the function evaluator for writing a RETURN_jmpbuf.

So whenever a BreakNode is evaluated, it longjmps to the BREAK_jmpbuf which magically puts control straight back into EvalLoopNode, regardless of how many other stack frames are in the way.

There is a small amount of book-keeping to make sure nested loops and nested function calls unwind correctly, but overall I think it is quite elegant.

I reused the longjmp implementation from the kernel. All it does is that setjmp copies the stack pointer and return address into the jmpbuf, and longjmp restores them. The result of a longjmp is that the setjmp call returns a second time. The first time (when the jmpbuf is created) it returns 0. When returning because of a longjmp, it returns the value that was passed as an argument to longjmp (1, in the example above).

Benchmarking

Using the same benchmark program as last time, we can add "SLANG (interpreted)" to the chart:

And we see that we get the best of both worlds! The growth rate as N increases is comparable to that of compiled SLANG code, but the fixed overhead is comparable to that of Lisp, because we don't have to waste time on the compiler and assembler. The runtime is almost constant in this example because it is dominated by the fixed overheads.

We can get a better idea of the differences between the compiled and interpreted SLANG implementations if we compare the overall runtime for a more complex program. This one naively calculates primality of all integers from 2 to N:

And now we see that the interpreted code only becomes a worse proposition than the compiled code once the overall runtime exceeds about 45 seconds.

In general I've observed that the interpreted code runs about 2x-4x slower than the compiled code, but we get to save spending 20+ seconds on compilation.

In my Advent of Code video, the solution for part 2 ran in 1m35s in the interpreter. The compiler took 52 seconds and the compiled code took 38 seconds, for a total of 1m30s for the compiled version, so the compiled version won, but only slightly. On the sample input the interpreted version is a big win.

I think this is actually a much better outcome than Lisp would have been, even if the Lisp interpreter was fast, because it means for any given program I get to decide whether I want it to start up quickly to test small changes (use the interpreter) or whether I want it to run as fast as it possibly can on larger inputs (use the compiler), without having to make any changes to the source file.

Memory overhead

Speed isn't the only reason to prefer the compiler. The interpreter is the largest SCAMP binary yet made, weighing in at almost 25K words (the next biggest is the assembler at 20K). Loading the interpreter basically halves the size of your Transient Program Area before you've even started, and you also have the size of the AST on top of that. For quick tests on small inputs that's not usually a problem, but it is something to bear in mind.

You can play with it in the emulator. Just type lisp at the shell to get into the Lisp REPL.

(Note that I think I recently made some bugs; it seems to try to evaluate spurious code after it completes evaluating larger forms. I'm not going to bother fixing it because I'm probably not going to use it, but if you play with it and think it acts funny... that's why).

Types and garbage collection

Almost everything in Lisp is made of "cons cells". A cons cell is just something that has 2 fields, that can each point to either NIL (the empty list), or to another cons cell. The 2 fields are called car (contents of address register) and cdr (contents of decrement register) due to the IBM 704 which Lisp was originally created for in the 1950s.

In practice we want to be able to store things other than cons cells (e.g. we want to represent integers without some complicated encoding made out of cons cells). For this reason I have established the convention that if the car field is nonzero, but less than 256, then it is a type identifier rather than a pointer to a cons cell. This is perfectly safe because the first 256 addresses in SCAMP memory are the bootloader ROM, so there can never be actual cons cells created at those addresses. Where the car field indicates that a cell has a non-cons-cell type, the cdr field is specific to that type. For integers it contains the raw integer. For strings it contains a pointer to the string, etc.

type	id	notes
NIL	0	used for empty list and "false"
SYMBOL	2	cdr points to interned string containing symbol name
INT	4	cdr is raw integer value
BIGINT	6	unimplemented
VECTOR	7	unimplemented
STRING	10	cdr points to string; only string constants supported for now
HASH	12	unimplemented
CLOSURE	14	cdr points to cons structure containing function body, argument names, and parent scope pointer
BUILTIN	16	functions implemented in SLANG, cdr is SLANG function pointer
PORT	18	essentially a boxed file descriptor
CONTINUATION	20	not accessible within the Lisp programs, just used for trampolined style
MACRO	22	similar to CLOSURE, except the value returned is then itself evaluated (this is not the correct way to implement Lisp macros, because you pay the macro execution cost every time you call it, instead of only once)

My Lisp interpreter uses a mark-and-sweep garbage collector. In order to be able to "mark" a cons cell, we need some way to store 1 bit of information in the cell. Given that each cons cell contains 2 fields, I established the convention that they are always allocated at even addresses. That way, any pointer to a cons cell is always an even number. If we also make sure that all of the type identifiers are even numbers, then every value that can legitimately be stored in a car field is an even number. We can now use the least-significant bit of the car field to let the garbage collector mark the cells that are accessible from the environment.

So the garbage collector looks in the current scope, and every scope in the call stack, and recursively marks all accessible cons cells as "used". Having done this, it sweeps through the arenas that cons cells are allocated in, frees any that are now unused, and clears the mark bit from cells that are still used.

The garbage collector is invoked at the top of the EVAL() loop if there are fewer than 100 free cells. That ensures there are always enough cells to do the work of one EVAL() iteration. This is an idea that I got from ulisp's garbage collector.

Before I came across the ulisp example, my first idea was that I would invoke the garbage collector from cons() if ever anything tried to allocate a cell and there were 0 free cells available. The problem with that approach is that some parts of the interpreter need to construct objects consisting of several cons cells, and it's bad if the garbage collector frees some of the cells halfway through constructing an object that isn't yet linked into the program's environment. Having a single defined place that the garbage collector can be invoked from makes reasoning about its semantics much easier.

Trampolined style

The natural way to write the interpreter is to make EVAL() recursive. Unfortunately this means you need arbitrarily-large amounts of stack space in order to support arbitrarily-deep recursion, and we don't want to have to make a commitment ahead-of-time as to how much space we allocate for the stack and how much for the heap.

My solution for this was to make EVAL() iterative, and store the Lisp program's "call stack" in a linked list made up of cons cells. This is quite pleasing in many ways because it helps to ensure that every reachable object is reachable from the cons cells in the program's environment.

Some parts of the interpreter need to call EVAL() on some Lisp form, and then do something with the computed value. The solution for this is to push onto the call stack a "continuation", which is just a SLANG function pointer and some associated state. Every time EVAL() finishes evaluating a Lisp form, it pops the calling continuation off the stack and passes control to it. I found out later that this is called "trampolined style". It is quite a flexible way to handle control flow, and for example it makes tail-call optimisation very easy.

Unfortunately it makes for control flow that is extremely difficult to understand. But I think it's just complicated because it's complicated. It's not obvious that there is an elegant way to implement it.

Benchmarking

I wrote a short program to test primality of an integer, in both SLANG and Lisp. (It does it the stupid way: just test for divisibility by odd numbers up to sqrt(n)).

Here's a chart showing overall runtime for both the SLANG and the Lisp version, with the time for the SLANG version including the time it took to run the compiler, and the time in all cases being measured from hitting enter at the shell prompt to getting back at the shell prompt (i.e. including process switching overhead, which occurs twice if we need to run the compiler).

The time to run the SLANG program is dominated by the time it takes to compile it, in all tested inputs.

You can see that even with compilation time included, running the Lisp program takes longer as soon as we get up to about N=3000. In other words, if you have a program that does ~30 iterations of a trivial loop, you're better off writing it in SLANG even though it takes 20 seconds to compile.

If you subtract out the compilation time and process-switching time, then the largest input I tried takes about 60 seconds in Lisp versus 0.2 seconds in SLANG. So the Lisp interpreter is about 300x slower. I put this down to the complicated trampolined style, and the fact that every operation involves allocation of multiple cons cells.

What's next?

I'm still after a better SCAMP development workflow. My main issue is that it takes too long to compile a program to test small changes.

I think I've ruled out both Lisp and my interactive SLANG REPL that compiles each line of code as it's typed. Lisp because it's too slow, and the SLANG REPL because it's too much of a footgun.

Maybe the next idea is to write a SLANG interpreter. That way I could run the program in the interpreter on small inputs, and only when satisfied that it is correct do I need to invoke the compiler in order to make it run faster. This has the benefit that we don't need to pay the compilation time on small inputs, but we get to use exactly the same program to get the fast runtime on large inputs.

Probably the way to do this is to rip the parser out of the compiler, have it construct an abstract syntax tree, and then interpret the AST with a straightforward recursive evaluator. I think the overhead of the trampolined style is too much to be worth paying, and besides SLANG code doesn't tend to recurse anywhere near as aggressively as Lisp code.

Unlike with the Lisp interpreter, I wouldn't try to make it "safe". SLANG code is inherently unsafe. I think if the interpreted program tries to do something that would run off and crash in the compiled code, it's perfectly fine for the interpreter to run off and crash as well. That means we don't need the interpreter to waste time on type checks and bounds checks. It'll just straightforwardly do the same thing that the compiled program would do, for example if you try to call an uninitialised function pointer.

The feet are ideal for this sort of thing, because they're the only part of your body that has any sensible degree of dexterity while still being invisible to casual observers.

There is prior art for phones taped to legs and some sort of TV remote control (?) and lots of cases of just going to the toilet and looking at chess positions on phones, but I think Sockfish may be the first hands-free method that does not require third-party assistance.

Each shoe insert has two force-sensing resistors and one vibration motor. The force-sensing resistors are used as buttons to allow me to input my opponent's moves. The vibration motors are used for haptic feedback of accepted button presses, and to communicate the engine's moves to me so that I can play them on the board.

On Tuesday evening I finally had a chance to deploy Sockfish in a real game against an unsuspecting opponent! Owen is quite a bit better at chess than I am. I talked him into playing a game with time control of 15 minutes per side, which is longer than the blitz & bullet that we normally play, but necessary to give me enough time to type the moves with my feet.

First game

Owen was very confused about why it took me 20 seconds of intense concentration to decide on my very first move. He eventually surmised that I must have "revised" and was concentrating hard to make sure I remembered the theory. In actual fact I was concentrating very hard to make sure I understood Sockfish's outputs correctly and gave my inputs correctly! I found that I concentrated much harder operating Sockfish than I do when I'm playing chess the hard way. Maybe I'd play better if I concentrated harder.

It was all going well until we reached a position where Sockfish was telling me to make an illegal move. The game (as understood by Sockfish) proceeded as https://lichess.org/cmcYJF7G (with me playing black). It wanted me to play 20... Qxe6.

But on the real board the position was different. Ruari was spectating, and already knew about the ruse, so he had contrived to record parts of the game. From his video, the real position was this:

The difference from the Sockfish version is that my g pawn has moved to g6, and my light-squared bishop has not captured on g2.

I think what must have happened is that when the computer asked me to play Bxg2 I instead played g6. This is not as crazy as it sounds: normally Sockfish tells me the starting square and destination square for the move it wants me to make, but since only one piece could move to g2, Sockfish only needed to communciate the destination square. So it sent g2, but I heard g6. This is because a "2" is transmitted as 2 short pulses of vibration on the right foot, and to save time a "6" is transmitted as a long pulse (representing 5) followed by a short pulse. I simply mistook the first short pulse on the right foot for a long pulse, and assumed I should play g6.

Maybe this diagram of the vibration pulses will help (or maybe it won't):

Anyway, with Sockfish now out the window, I had to continue on my own.

Instead of illegally capturing the queen on e6 (which I judged that I would be unlikely to get away with), I blocked the check with the rook. This was foolish in the extreme! The rook is simply captured with no compensation. In my defence I had been concentrating so hard on communicating with Sockfish that I scarcely had any comprehension of the actual chess position. We soon ended up at this position and I resigned:

Second game

Thankfully, the jig still wasn't up, and Owen agreed to a rematch.

Ruari filmed almost the entire game this time. I remember being paranoid that it would be obvious that I was fidgeting, but in the video it just looks like I'm thinking really hard. I don't think anyone would guess that I was using trick shoes.

At one point during this game I made an input mistake, but I noticed it before it became a problem! I was seen to mouth a swear word, think hard for 30 seconds, start laughing, and then ask Owen to remind me about the previous couple of moves. It's a wonder he didn't suspect anything.

After spending 70 seconds secretly undoing and redoing moves with my toes (all while trying to act natural) I eventually convinced myself that I had recovered the position and proceeded, with an audible "phew!" and an announcement that I needed to stay focused.

Unfortunately, a couple of moves later, disaster struck once more! Sockfish again asked me to play an illegal move. The position in Sockfish was this:

But the position on the real board was this:

Basically 27... Qxe1 and 28. Rxe1 had gone missing from Sockfish's version while I was undoing and redoing moves. I must have undone 2 ply too far and failed to reconstruct it. In this position Sockfish was asking me to play Rd8+, but since I didn't have a rook on the d file, this proved challenging. I announced that I was now "off-piste" (without explaining what that meant), and started playing manually.

Fortunately, being a queen up, the game pretty much won itself, and Owen resigned a few moves later. Not exactly the great success that I had in mind for Sockfish, but at least it was a win! 50% of the time, it wins every time.

Grand reveal

Immediately after winning the game, I explained the ruse to Owen. His first reaction was incredulity, until I showed him the shoe inserts and computer. Overall he took it pretty well.

I told him that I was planning to recruit a "plausibly-good" chess player to use the shoes to win the world championship, but he considered that this may "bring the game into disrepute" and said he would definitely dob me in. I think I'd be more interested in the Netflix adaptation than the real-life story anyway.

Details

In my pocket is a plastic box containing the Raspberry Pi Zero, a 4-channel ADC for reading the force-sensing resistors, some transistors to switch power to the vibration motors, and a trimpot for each vibration motor to adjust the strength. Without the trimpots they are very loud and might give the game away (so to speak) prematurely. This is what's inside the plastic box:

There's also a USB powerbank in my pocket to run the Pi.

The Pi is running a Python script that reads from the GPIO pins and writes to stdout, which is consumed by a Perl script that reads button presses from stdin, reconstructs chess positions, and runs Stockfish, and then writes vibration pulses to its stdout, which are then sent to GPIO pins by a second Python script.

(Why this crazy architecture? I am happier writing the actual program in Perl, but all the Raspberry Pi GPIO examples use Python... it was just easier this way).

This whole contrivance is running inside screen with -L so that the output is logged for perusal later. It's started by /etc/rc.local when the Pi boots up.

If you have the shoe inserts plugged in while the Pi is booting, then one of the vibration motors stays stuck on for about a minute, because it turns out that the GPIO pin that drives it is initialised to +5v while the Pi boots up. Doesn't really matter, but you can feel the vibration get weaker over the course of the minute as it heats up and the resistance increases.

I used a soldering iron to poke a hole in the cargo pocket of the trousers so that the wires can run down the insides of my trouser legs.

Originally I was using 4-pin TRRS connectors to connect the Pi to the shoes, but I found these were very flakey and unreliable. I replaced them with 4-pin waterproof ebike connectors which work much better and are not significantly larger.

At first I was printing the shoe inserts in PLA, which is too stiff and makes it very difficult to walk. I replaced these with flexible TPU plastic, which worked much better. I also started out with ordinary microswitches, which got damaged when I walked on them, and replacing them with the force-sensing resistors solved this problem, with the added bonus that I can now adjust the activation force in software.

Jez pointed out that Sockfish is reminiscent of the Eudaemons' roulette-beating shoe computer, which is a very impressive project considering the primitive electronics available in the 1970s.

Sockfish-ng

Don't tell anyone, but I already have plans for a better version of Sockfish that will be much harder to detect!

This proof-of-concept only needed to fool my mates, in a pub, for the duration of 2 games. To win the world championships we're going to have to get much more serious.

I'm thinking we'd integrate the Pi and all the other electronics onto a single PCB, instead of having a normal Pi plugged into 2 pieces of stripboard. We'd put this compute PCB in one of the shoes. Each of the shoes would have its own battery and Bluetooth (or similar) radio inside, and all of the electronics would be hidden under the in-sole which would be glued down. Even strip-searching the player would not reveal any secrets.

The interface would work similarly to how it already does, maybe with better distinction between a long pulse and a short pulse, except the communication from the secondary shoe would go over the radio instead of a wire.

You'd probably have to X-ray the shoes to find out something is up. If that is a realistic threat then I think the plan is to have the player walk in wearing ordinary shoes, have someone else smuggle in the hacked shoes through a different route, and have the player swap shoes once he's already been scanned.

I envision a handheld device, maybe a bit bigger than a scientific calculator, and with a bigger screen and fewer keys. Here's my concept sketch:

I want you to imagine that instead of a crude freehand sketch, you are looking at a beautiful 3D rendering with maybe a shiny metal case and some cool diagonal air vents on the side.

Then we can imagine that the keys have satisfying Cherry MX blue switches, the display is maybe e-Ink, or something else equally cool-looking (because looking cool is most of the point of a cyberdeck), and there's an integrated battery with USB charging. I also thought it would be cool to have a lambda calculus reference sheet that slides into the back of the case.

The runtime could have some way to detect when the final evaluation of an expression is a Church encoding and pretty-print the output, and maybe it could accept input over USB serial to save me from having to type everything out perfectly every time I want to test it.

There are just a small handful of problems.

Lambda calculus problems

It turns out that programming directly in lambda calculus is really hard. I actually don't know how to do it yet, so the stuff I wrote on the display in the concept sketch may be meaningless. (I think (λx.λy.x)x means something like "a function of x that returns a function of y that returns x, applied to the symbol x", which would yield a function that ignores its argument and always returns our symbol x?).

You can add useful features to lambda calculus (such as support for numbers, and functions that take more than one argument) but those things aren't strictly necessary. And besides, if you carry on down that road then you just end up at Lisp. And noone wants a Lisp machine cyberdeck.

And to be honest, being difficult to program is a big part of the appeal for me, so maybe this isn't such a bad thing.

I was struggling with how to handle variable names, but I think what I've settled on is that the only letter available is x, and you make other variable names by appending apostrophes, so you can have x, x', x'', etc. (copied from the "austere" version of typographical number theory). This way we only need 2 keys to express arbitrarily many variable names, with no ambiguity about the end of one name and the start of the next.

Display problems

Most of the electronics and software is easy enough (take input from the keypad, handle line editing, evaluate lambdas), so everything only really has to fit around operating the display.

I have a Pimoroni "inkyWHAT" 400x300-pixel e-Ink display lying around unused.

The inkyWHAT is intended for use with a Raspberry Pi. I'd rather not drive the display with a Raspberry Pi though, that seems a bit too heavyweight. I'd rather this thing wasn't secretly running Linux underneath.

I also have a Raspberry Pi Pico lying around, which might be suitable for the cyberdeck.

So far I have tried to program the Pico using MicroPython in Thonny, but I found it to be a pretty bad experience. Firstly because the Python library for driving the inkyWHAT is only compatible with a normal Raspberry Pi, which for a Pico means ripping out all the parts that use numpy and RPi.GPIO, and rewriting them to suit MicroPython.

Secondly I found that either Thonny or MicroPython or both are pretty buggy. There seems to be only a very weak link between the code shown in the IDE and what is running on the hardware. The code you type in your main file seems to be reloaded OK whenever you click the green "RUN" button, but any code you put in other module files seems to be loaded only the first time you use that module, and subsequent changes have no effect until you unplug it and plug it back in. Even the big red "STOP" button doesn't do the trick.

After spending a couple of hours (unsuccessfully) porting the inkyWHAT display driver to MicroPython for the Pico, I concluded that I was going too far into yak-shaving and gave up.

Probably I'd just use a Pi Zero and tolerate the fact that it takes a couple of minutes to boot up. Or maybe stick with the Pico but write everything in C instead. The Pico's C SDK installation instructions don't look too complicated, but they are more complicated than "launch the Arduino IDE", which is what I'm accustomed to.

So the question is: do I really want a lambda calculus cyberdeck?

If you don't know: Doodle used to be a very simple tool for coordinating social occasions. One person would create a "Doodle poll" listing proposed dates, and everyone else would tick the dates they could attend, so you can select an optimal date for the event. It beats the game of "can you make this date?", "no, I have another thing on, how about the day after?", "Fred can't make that, what about this other date?", and so on.

Unfortunately, like so many previously-great products, Doodle has slowly got worse over time and is now your standard corporate bloatware. I guess when you have a good product and people like it, the natural response is to change it until they don't. Doodle used to just do dates, but it now seems to be more focused on times-of-day, which hinders the original purpose because you can now only see 7 days at a time instead of an entire month. The straw that broke the camel's back is that they now require email addresses from all poll respondents. Doodle is now shit. They should have left it alone.

So Eldood is what Doodle used to be. It's on github.

The backend is a Mojolicious application that stores data in a SQLite database.

At first I was going to make a table that joins a respondent to a single date and a poll, but in the end I went with "CSV-within-a-column" because normalising it seemed more complex for no reason. Each response is a single row in the responses table, with a column called dates that has values like 20220630,20220701,20220702.

To provide the "ifneedbe" option, dates in this column can have parentheses around them. It's not pretty but it is quick and easy.

And, crucially, it is better than Doodle.

And Matt pointed out that the universe actually contains many surprising things. Electricity, magnetism, gravity, fire, light, planets, trees, boats, computers, and so on. These are all very strange and surprising things that you probably couldn't imagine if you didn't already know about them, and yet they exist in our universe with no trouble at all. Why should consciousness be any different? And yet consciousness does feel different. All of the other things happen, apparently, completely mechanically, even if the mechanisms are complicated. But consciousness has a mind of its own.

What do I mean by consciousness?

By "consciousness" I don't mean other related things like emotions, feelings, intelligence, or decision-making. I think those things can be explained without any "observer".

I struggle to put into words exactly what I do mean, but I think it should be obvious to anyone who is actually conscious. I am aware that I exist. I think therefore I am. I have an internal experience, in which I am the subject of things that happen to me.

Sometimes when I talk to people about consciousness they try to argue that it is an illusion, and that we're all tricked into thinking we're conscious when we're actually not. I find this position self-contradictory. Consciousness isn't the illusion, consciousness is the audience. Anyone who has ever been conscious knows that being conscious is different to not being conscious.

Arguing that there's no such thing as consciousness is like arguing that there's no such thing as hunger. "Oh, people just say they're hungry when they haven't eaten lately, but nobody actually feels hungry". But if you've ever felt hungry in your life then you know that this is false. The same with consciousness: the only way you could seriously believe that consciousness is an illusion is if you have never experienced it.

And that is a distinct possibility! A p-zombie is a hypothetical person who looks and acts and seems just like an ordinary person, except they have no consciousness. There is no internal experience. There is no awareness. A p-zombie takes the same sorts of inputs as a conscious human, and produces the same sorts of outputs, and for all intents and purposes appears conscious, but is actually not, and there is no way to tell from the outside.

It is entirely possible that some (or many, or most!) of the people in the world are p-zombies. Solipsism holds that everyone in the world is a p-zombie, apart from me.

It might sound egotistical to imagine that I may be the only truly conscious being ever to exist, but I don't think it's that far outside the realms of possibility. After all, I can hear my thoughts and I can't hear yours.

Personally I prefer to live in a world in which other people are real than one in which they are not, so I discount Solipsism out of hand. Even if I am the only truly conscious being in the universe, I think I get more out of it by pretending the NPCs are real.

What do I mean by computable?

I thought this was a pretty settled question, with an answer something like "able to be calculated by a Turing machine", but it turns out there is a field of hypercomputation which studies computational models stronger than that of a Turing machine.

But by "computable", I mean "able to be calculated by a Turing machine". For consciousness to be computable, I mean that it is possible for a Turing machine to create consciousness.

I don't merely mean that it is possible to create a convincing simulation of a consciousness that gives plausible outputs for any given inputs. That would be a p-zombie. Anyone who is conscious knows there is more to consciousness than just saying you're conscious.

Argument

Premise 1: Everything in the universe is explained by the laws of physics.

By "laws of physics", I mean the complete set of laws that govern the universe. This premise seems almost tautological.

I think the only way it might be disputed would be if you argue that there is no set of laws governing the universe, and literally everything can happen, and does happen, and no laws apply. That the only reason we seem to have laws on a macro scale is some sort of constructive interference between all the different ways the universe can do everything all at once.

Premise 2: Physics is computable.

I'm not sure how to argue this one. Classical physics is computable. Relativity is computable. Quantum mechanics is computable (even without quantum computing) for whatever finite degree of precision you want. What would non-computable physical laws even look like?

Premise 3: Consciousness exists within the universe.

The best arguments I know of in favour of this premise are:

• everything exists within the universe; that's what it means to be the universe
• anaesthetics exist within the universe and are (seemingly) effective at suppressing consciousness, so the consciousness must also exist within the universe

Alternatively, it could be that the universe is more like the world in a video game, except the player of the game has their senses hooked up directly to the game at birth, so that they have no way to receive sensory input from anything outside the game universe. As far as such a player would be concerned, the video game is all that exists. They would be just as baffled as we are about how consciousness can arise in such a primitive world.

You could even imagine that anaesthetics exist within the game, except whenever the player in the game receives anaesthetic, the "real life" player is also injected with anaesthetic (by a robot or diligent minder), such that it appears to the player that the anaesthetic within the game is fully functional. Arguably it is.

The "video game universe" idea doesn't even require you to assume that everyone else is a p-zombie. Multiplayer games exist.

Conclusion: Consciousness is computable.

If you accept all 3 premises, then I think you have to accept the conclusion. However, the conclusion is obviously absurd. It seems obvious to me that no amount of computing can create the sensation of existing as a mind.

So I think this suggests that one of the premises is false, which also seems surprising, but maybe not quite so much? At least there are 3 times as many places for the surprising part to be hiding.

Either minds have something that computers don't, or they don't. Neither possibility seems correct.

The reason to prefer a REPL over the edit-compile-test workflow is that it saves wasting time on recompiling and rerunning the entire program every time a small portion changes. In principle you can update a function definition, and then re-compile and re-run that function, without having to recompile the rest of the program, or repeat (say) parsing of the input file.

I think it would not be too much trouble to make an environment that lets me type a line of SLANG code at the command line, and then compiles the code, loads the compiled code into memory, and executes it, "inside" the process of the development environment. This wouldn't be compiling a program as such, it would just be a tiny amount of code each time. As long as we tell the compiler the addresses of all of the known globals, and what the start address of the code will be, this seems perfectly viable.

Once we have an interactive environment where we can type in single statements and compile and execute them, there are a number of interesting things we could do:

1. Notice declarations of globals and allocate them manually, so that they can be reused in later statements.
2. Remember the source for each function definition so that it can be edited later without having to type it out from scratch.
3. Save every input line in a history file so that the history can be manually edited down and turned into a standalone program.
4. Save the process binary out to a file that can be executed to get back to the current state.

I have already got a rudimentary implementation just to find out how well it works. It's called "RUDE", for something like "RUDE Ultimate Development Environment". It's only about 350 lines of code. It can notice global declarations, compile and execute code, and save out the project in both source and binary form.

You can try it out in the online SCAMP emulator if you want, just run rude at the shell to get into RUDE.

One advantage over FORTH is that function calls in SLANG always have an extra layer of indirection. This only exists to simplify the compiler (there's no need for a distinction between ordinary function calls and function pointer calls if every function is a function pointer... Galaxy brain), but it means when we update the implementation of a function, all of the callers of that function automatically use the new implementation. In FORTH, callers of a function still use the old implementation until they are in turn recompiled, which is bad because you can fix a bug and find that your fix is not actually in use until you track down everything that calls the function that contained the bug.

Current state

Here's an example RUDE session with what I have so far (in the emulator, running about 20x faster than real hardware):

(All of the "... 1st pass ..." stuff is spam from the compiler and assembler.) The important things to note in this session are:

1. Printing "Hello, world!" using printf. The call to printf is compiled, then executed. The text "Hello, world!" appears on the console, and we see the return value of 14 from printf (the number of characters written).
2. Declaring a variable "n" and initialising it to 0. The assignment is compiled and then executed. There is no real return value, so the displayed return value of 20960 is some spurious memory address.
3. Declaring a variable called "f" and initialising it to point at a function that increments "n" and then prints out the value of "n". The function definition and assignment is compiled and then executed, which executes the assignment to "f" (but does not execute the function body).
4. Calling f(). We can see that "n" is incremented with each call (and we get the return value of 20 from printf, by accident).
5. Exiting RUDE (typing ^D to give EOF on stdin). We now have files project.sl and project. project.sl contains a source listing of the session, and project contains a copy of the process binary from memory, taken at runtime after the latest command line input.
6. Executing ./project takes us back into RUDE (but with no "RUDE AWAKENING" message, because we're dropped straight back to the REPL, we're not restarting RUDE), with the in-memory state exactly as it was when we exited! Good magic.
7. Calling f() proves that both the function implementation and the value of "n" are retained.

Killer problems

My current implementation, although it proves the concept (maybe), is not workable for several reasons:

Compiling takes too long

It turns out that it takes over 30 seconds (at 1 MHz) to compile the empty string. This means almost all of the compilation time, for typical statements, is wasted on overhead which is exactly the same every time.

What is this overhead?

• preparing source inside RUDE: ~8 seconds
• context switching in the kernel: ~7 seconds
• compiler: ~7 seconds
• assembler: ~10 seconds

(Assuming I didn't blunder the profiling).

When we're compiling "the empty string", we're not just compiling the empty string. RUDE also has to provide names and addresses of globals, of which there are a bit over 200 in a blank environment (mostly the functions in the standard library). A lot of time is spent formatting and writing these 200 names and addresses (in RUDE) and reading and parsing them (in the compiler and assembler). I think that accounts for most of the overhead.

The rest of the overhead is context switching in the kernel, which means saving the parent process (RUDE) out to disk, and reading child processes into memory (compiler and assembler). I've spent quite a lot of time optimising context switching in the past so I expect there's not low-hanging fruit here, short of reducing the number of context switches. Currently we have RUDE swapped out, the compiler swapped in, RUDE swapped back in to do a tiny amount of work, RUDE swapped back out again, the assembler swapped in, and RUDE finally swapped back in again, which is about 6 lots of reading/writing a process to disk.

Unknown binary size

When we compile a statement, we need to malloc() some space to put it in, and we need to tell the assembler what the starting address will be (there's not yet any sensible way to create position-independent code).

Perhaps you can see the problem here: we don't know what address the code will be placed at until after we've allocated space, and we don't know how much space to allocate until after we've generated the code, which we can't do until after we know where it's going to be placed.

Currently RUDE solves this by always allocating 1000 words for every statement. This is bad, both because it is way too much in the normal case and pointlessly wastes memory, but also plausibly because it could be too small in some cases.

Memory leak for almost all statements

But it's worse than just temporarily allocating too much space for each statement. We're permanently allocating too much space for each statement!

RUDE has no knowledge at all of what the statements are doing. The only special case it currently supports is that statements beginning var are declarations of globals, which require special handling so that the addresses of the globals can be tracked.

Lots of statements are completely ephemeral. For example, printf("Hello, world!\n", 0) doesn't contain any data that needs to persist after it has finished executing, so it would be perfectly fine to free() the memory that we put the code in, after it's been executed. But without peering inside the code (and, in turn, peering inside printf() to work out whether it is expecting to keep hold of the string), there's no way to know that the statement doesn't contain static allocations that need to stick around.

Statements that include function definitions, strings, or array literals contain static allocations, and if executing the statement results in a reference to one of the aforementioned allocations being kept around somewhere, then it is not safe to free (all of) the code space because that would result in freeing the memory backing the corresponding function, string, or array literal.

Currently RUDE just doesn't free anything. The compiled code for every input statement stays in memory forever, even when no references to it remain. Incidentally, this is the same approach that simpler FORTH implementations take, as far as I can tell. They just don't even implement free(), all allocations take the "HERE" pointer, and advance it ready for the next allocation. That said, FORTH doesn't have to allocate code space for most input statements, because they are executed without being compiled.

Killer solutions

We need to solve all of the above 3 "killer problems" in order to turn RUDE into a usable system. I haven't got a complete picture in mind yet, but I have the following ideas which might form part of it.

"Interpreting" simple statements

Simple statements could be interpreted directly inside RUDE. We already have the table of all known globals. If we did a tiny amount of parsing, to recognise simple literals and function calls, and throw everything else to the real compiler, then we could "interpret" simple statements by pushing function arguments onto the stack and then calling the function pointer that we look up in the globals table. We might save both execution time, because simple statements aren't compiled, and memory usage, because we don't need any memory to store code we're not producing.

I definitely want to do this, in the absence of some other great breakthrough that makes compilation effectively instantaneous (see below).

Put the compiler and assembler inside RUDE

The limit case of interpreting simple statements is to interpret all statements. The easiest way to do this would be to basically copy and paste the source of the compiler (and assembler) and put them both inside the RUDE process, so that we never have to context switch, and we already have the table of global names and addresses ready to hand.

This would be an ideal solution, apart from one thing: memory. Currently the RUDE binary is about 15 Kwords, the compiler is 18K, and the assembler is 20K. That comes to 53K, which is already too much to fit in memory (taking account of the kernel), let alone any code or data for the user program.

In practice it wouldn't be as bad as that because the library functions would not need to be duplicated. The library blob comes to 9K, so if we lose 2 copies of that, then we could expect putting the compiler and assembler inside the RUDE process to bring it up to 35K, which is still on the extremely heavy side.

This idea might be worth playing with anyway, because I can't think of any better way to reduce compilation overhead. But the memory cost is very high, it certainly wouldn't be workable for the larger Advent of Code problems. But maybe it could work if I develop in RUDE, get it to work on the sample input, and then turn it into a standalone program to run on the full input.

If everything were inside the one process, it might also make it easier to defer memory allocation until such a time as the length of the binary is known.

Give names and address to the compiler and assembler in a better format

Currently the names of globals are passed to the compiler as ordinary SLANG source, like:

extern printf;
extern getchar;
extern malloc;
...

These are just directives that tell the compiler that a name exists as a global but is declared somewhere else. It's just a way to say that it's fine to refer to these globals, even though the compiler didn't see them, because something else is going to tell the assembler where they live.

Similarly, they are defined for the assembler as ordinary assembler directives, like:

.d _printf 0x2f10
.d _getchar 0x2f0a
.d _malloc 0x2e8d
...

So that the assembler knows that any reference to _printf is actually a reference to 0x2f10.

The problem with this scheme is that it is designed to be easy for humans to write, and parsing it with the ordinary parser is very slow. Since this table is both generated and consumed by the machine, maybe it could be in a binary format which is faster to work with, and can be parsed with no validation or bounds-checking.

(I care about performance, not security. Fight me. Nobody's complaining that RUDE is a non-viable project because it's too easy for hackers to break into your SCAMP!).

Stop the compiler from validating names of globals

Rather than create a whole new format for passing globals into the compiler, we could just have a switch to make the compiler assume that all unrecognised names are valid globals. The downside is that typo'd names won't get rejected until assembly time, and the message will be slightly more confusing, but it would save us time in the compiler.

It's probably not worth doing because even if we saved 100% of the overhead in the compiler, and 50% of the overhead in RUDE, it would only knock about 7 seconds (25%) off the total time to compile the empty string.

Have the compiler directly output machine code instead of assembly language

Instead of having the compiler write assembly code, and then passing the assembly code to the assembler to turn it into machine code, we could have the compiler output machine code directly.

Reasons not to do this include:

• Opcodes can change whenever I change the instruction table, so any code that cares about opcodes needs a system to keep it updated, which is a hassle. (Although I don't do that much any more, so maybe it's fine to say opcodes are set in stone now).
• We'd still need some equivalent to the "2nd pass" of the assembler which touches up addresses that weren't already known during the first pass. Probably this would be mostly copy-and-pasted out of the assembler and into the compiler.
• It's helpful to be able to have the compiler driver (either slc or RUDE) pass in its own assembly language header and footer that can use labels instead of hard-coded addresses. Without a separate assembly step these addresses would probably have to be patched up in the binary, which is obviously more trouble and less robust.

But those are small problems.

The assembler accounts for the majority of the runtime in compiling normal programs as well as short statements in RUDE, so removing it could more than double the compiler throughput. So I probably want to do this.

Then it might be worth revisiting "Put the compiler inside RUDE" because we would save almost the entire memory cost of the assembler.

Split up lexing and parsing

The parsing framework I use for both the compiler and the assembler does not distinguish between lexing and parsing. Every input token to the parser is a single character. This introduces quite a lot of overhead because sometimes the same characters are examined many times until enough backtracking has happened to work out how to parse them. Historically this was tackled by first tokenising the input (so each character is only examined once) and then parsing the (much shorter) stream of tokens instead of characters.

I'm not sure how much help this would be. I have structured the parser in both the compiler and the assembler to try to minimise the amount of backtracking on common inputs. It's possible that separate tokenising would actually make performance worse because now instead of 1 stream there are 2 streams that need to be consumed!

Compile everything twice

To work out how much memory we need for the code, we could compile it once with a placeholder address, measure how large it is, make the allocation, and then compile it "for real". This would double the time spent on compiling, which makes it sound absurd, but in combination with some other solution that makes compiling very fast, it might be the easy way out.

In principle the size of the generated code can depend on the address it is going to be placed at (because some instructions have shorter forms for addresses in the top page and the bottom page), but in practice I think it would always be the same size, because we're not putting code in the top or bottom page.

Make realloc() shrink in-place

Currently realloc() always makes a new allocation, copies the data, and then frees the old allocation. But when you're reducing the size of an allocation, it's always safe to just shrink the existing allocation and return the unused space to the free space pool.

So we could just allocate a block that's way too large, put our code in it, and then shrink it to fit.

The only reason I haven't implemented shrink-in-place already is that almost every use case I had for realloc() involves growing an allocation to handle more data.

I think making realloc() shrink in-place is the best solution to not knowing the binary size ahead of time.

Generate position-independent code

I remembered that I have already written up my thoughts on how position-independent code could work for when I wanted to implement strace().

Basically it involves making a no-op function call, so that after it returns you have your Program Counter in r254, and then you know the global offset to where the code is running and can patch up all the addresses before running the "real" code. It's a bit of a hassle but would work.

So maybe I could do that and then defer allocation until after the code is already compiled? Probably a bad idea, it seems both more difficult to implement and more wasteful of memory than the "Make realloc() shrink in-place" idea.

Manual statement annotation

To solve the problem of wasting memory on side-effect-free statements, we could require the user to manually annotate "important" statements with an exclamation mark, so that they're not immediately free()'d after execution. This seems error-prone though, I'm not sure it's a good idea.

Ben pointed out that the interactive REPL system I'm describing is a bit like IPython. IPython annotates each statement with a number, and each statement sticks around until manually deleted. Maybe I could do something like that, where instead of deciding ahead-of-time whether a statement should be free()'d or not, the user just has the option to manually free statements that they no longer care about. This would also be a good way to manage the SLANG source history without having to manually delete dead-ends.

I don't really like it, but maybe it's not as bad as it sounds? And I don't have any other ideas for how to stop wasting memory, so maybe it's the best you can do?

Next

So I have a lot of ideas to be pondering, but still no complete picture for how RUDE is going to work.

I think the way to go is to start with the promising ideas that would be useful even if RUDE never works, namely:

• Have the compiler directly output machine code instead of assembly language
• Make realloc() shrink in-place

And then at least I've got some benefit out of it, and I can reassess performance and memory usage to work out whether I want to press on with RUDE or ditch it.

It can be used for holding small parts while drilling, filing, tapping, etc., or for transferring multiple parts between more substantial workholding arrangements (e.g. the vice on the mill and the vice on the bench) while maintaining their alignment, and probably for lots of other things.

The way it works is you tighten up the front bolt (which pulls the jaws together) until the clamp is grabbing the work, and then tighten up the rear bolt (which pushes the jaws apart, it has a small section at the end of the thread turned down to locate in a matching hole) to lever the clamp against the front bolt and increase the clamping pressure. The knobs have holes in so that an allen key or similar can be used to provide more torque to the bolts.

I made it out of hot-rolled mild steel bar stock, because it was the most convenient material I had available, so apart from the 2 bolts, this is what it all looked like before I started:

Knobs

Normally on a toolmaker's clamp the knob and the screw thread would be made out of a single piece of material, with the knobs knurled on the lathe. I don't have a knurling tool and have always found cutting long threads in thin material to be quite tricky, so I decided to make life easy for myself and use existing M6 bolts, with the knobs machined separately and pressed on to the heads.

This turned out to be a bad idea, because the knobs ended up wider than the clamp jaws. This means, for example, it's not possible to use my clamp to hold something perpendicular to the pillar drill table for drilling, because the knobs prevent the jaws from sitting flat. Fortunately it won't be any harder to remake the screws now than it would have been to make them in the first place, so I can always replace the screws if it bothers me.

The jaws are 12mm across, and the head on the M6 bolts is 10mm across. This sketch in SolveSpace shows that a 10mm hexagon is 11.55mm across at the widest point:

So if you want the knob to press onto a 10mm hex head, and you want it less than 12mm in diameter, then you'll have less than 0.25mm wall thickness, which seems not enough.

I made the knobs with a 5mm end mill, whereas I made the jaws of the clamp with a 10mm end mill. I found that the 5mm end mill gives a much superior surface finish. I'm not sure whether the 5mm tool is made better, or whether my feeds and speeds were better. I used Bryan Turner's feeds and speeds calculator for both tools. Maybe it is just down to the rigidity of the machine.

Flood coolant

I haven't got around to improving the drain on my tub yet, but I ran some of these jobs with flood coolant turned on, and it worked really well. It did a great job of keeping everything cool and blowing chips away from the work. Since I haven't fixed the drain, I did end up submerging one of the stepper motor connectors, and nothing bad happened, so it looks like the coolant is indeed non-conductive. I also haven't noticed any rust developing, so that's good too.

Here's a video of the finishing pass on one of the jaws:

I don't know why it makes a horrible noise when cutting in the Y axis but not in the X axis. It should have been taking the same width of cut on all sides of the part.

Draughting

I've been getting interested in hand draughting lately, so I did a drawing of the toolmaker's clamp jaws while I was halfway through making them. I'm not sure how legit it is to have one drawing specify 2 slightly different parts, but I decided it beats having to draw it twice.

The jaws I made don't exactly match the drawing, but they're quite close. They're about 96mm long instead of 100mm because of a small blunder (I forgot to account for the diameter of the tool when zeroing my coordinates, rookie mistake). And the supposed 8mm deep blind hole is actually a through hole, because of another small blunder: I started drilling the hole from the wrong side, and a blind hole from the wrong side doesn't work, so it had to become a through hole.

At this point I didn't have a sensible way of drawing circles, so the circles are drawn freehand. I think it's OK apart from the circles.

Here's a poorly-lit photograph showing my homemade draughting equipment:

The board is an offcut of kitchen cabinet material (MDF with a very smooth paint job). The 30° triangle is 3D-printed, and is only there to assist with the isometric view. The T-square has a 3d-printed board edge follower on the left hand side, and the transparent plastic ruler is made out of a piece of broken perspex from a picture frame. The markings are engraved with a drag engraver on the CNC router and then coloured in with Sharpie to make them black.

It felt a bit absurd to draw a 30° angle in CAD in order to 3d print a triangle to allow me to draw 30° angles on paper, but it was genuinely the quickest way I could think of to create a 30° triangle.

Since then I have also made a simple compass, so I might be able to draw better circles next time:

On the underside there is a pin from a pogo pin pressed in with a soldering iron to provide a sharp point. You just adjust the radius with the wing nut, push the point against the paper, put the pencil in the hole, and swing it in a circle. I haven't used it in anger yet, so time will tell whether this is easier or harder to use than a normal compass. It's based on John Heisz's compact compass design.

And I have some vague plans for an ellipse-drawing tool to help me draw circles in the isometric view. There is prior art for ellipse-drawing tools, but I'm not aware of one that maintains the aspect ratio as you adjust the size. The ones I've seen have 2 adjustments that you have to adjust together to get the ellipse shape you want, but for circles in isometric view, you always want the same aspect ratio. So maybe I can invent a new thing here. Or more likely it already exists and I just haven't found it yet.

The machine is now installed in its tub, with the control cabinet on the wall and the PC on a little desk.

I've also restrained the cables with a handful of little 3d-printed clips, so that they can't get snagged:

Motor

I've wired up the spindle motor with an analogue ammeter for a crude "spindle load" display:

(video demo)

I found the spindle load display very useful for "manual" drilling, in the absence of a manual quill. I can modulate the Z feed rate using the jog pendant (see below) while watching the spindle load, to maintain cutting load without overloading the motor. It's also helpful generally for knowing how close a cut is to the limit of the motor. It seems to be fine with anything below 2 amps, but starts to stall if it exceeds 2 amps for more than a few seconds.

I've fitted one of those magnetic tachometer kits so that I can get a spindle speed display:

For the time being I've just stuck the magnet to the drawbar magnetically. It hasn't flown off and hit me in the face yet.

PC

I bought and installed the cheapest non-Nvidia GPU I could find on eBay, and it has solved the problem where OpenGL usage on the integrated GPU would crash the machine. That means I now have access to a much wider range of LinuxCNC screens, and I'm now on Gmoccapy instead of Qtaxis.

Surprisingly, the LinuxCNC screens seem a lot less mature than UGS Platform does. They all have weird bugs and strange UX. In Gmoccapy:

• If you edit a G-code program, save it, and then try to return to the main GUI, it asks you if you're sure you want to exit with unsaved changes. Even though you just saved. The problem is that there are too many abstraction layers and the save button has no way to communicate the fact that the text has been saved, either to the textbox or to the GUI.
• I have to manually click "home all" to unlock some parts of the GUI even though homing is a no-op on my machine.
• "Restart from line" does not reset to 0 after restarting. "Restart from line" is a useful feature that allows you to restart an aborted program from a line that you deem sensible, without having to manually delete all of the lines that come first. The problem is that a.) once you have restarted a program from a given line, any time you start a program, it will always start from the line number you last entered, even if the program completely finished and you've opened an entirely new one where carrying over the line number makes no sense, and b.) the starting line number is not displayed anywhere, so you just have to remember to manully reset it to 0.
• There is no cycle time estimate. UGS Platform has a really simple cycle time estimate based on the assumption that every line of G-code takes the same amount of time to execute. This is fine and useful enough, but Gmoccapy just has nothing.

I looked at fixing some of this, but the code running on my PC seems to be significantly different to the code on github, so I guess improvements are being made, and my problem is just that the latest release is quite old.

Probe_basic looks to be a really nice GUI for LinuxCNC, and even has a "conversational" interface where you can for example ask it to do simple facing operations without having to create a G-code program yourself. Unfortunately it requires a display of exactly 1920x1080 pixels. The display I'm using is only 1024x768 so it doesn't fit. I tried a (very bad) spare 1920x1200 display that I have, but Probe_basic seemed to scale itself to fit the 1200 pixel axis instead of the 1920 pixel one, so some of the horizontal space was hanging off the edge of the screen. I gave up in the end and reverted to Gmoccapy and the 1024x768 screen.

Jog pendant

Before starting a program, you need to set the coordinate system so that you machine material out of your work piece, instead of the air or the vice. This involves manually "jogging" the tool around to find a reference position and then setting the coordinate system from there. Unfortunately my computer is the other side of an opaque laundry basket, so when I can reach the keyboard I can't see where the tool is.

For this reason I have made a "jog pendant" that I can hold while standing closer to the machine.

It is just a 9-key macro keypad, inside a 3d-printed case. Pressing "Step" sends an "i" keypress which is what Gmoccapy uses to change the step size. Pressing "X left" sends a left arrow keypress, which moves in the negative X direction, and so on. "Rapid" is a shift key. Holding shift makes Gmoccapy jog at full rapid speed instead of the configured jog speed.

Making chips

After I first got the motor wired up, I clamped down a piece of aluminium and ran an end mill back and forth just to get a feel for it.

(I knew the workholding was poor, I just didn't have anything better available.)

I was not very impressed with the power of the motor! It kept stalling on relatively shallow cuts in a relatively soft material. At first I thought I was definitely going to have to get a better motor, but I think my main problems in that video were that the drawbar was loose (you can hear it rattling), the workholding was inadequate, the spindle speed was too low, and the cutting tool was blunt. It works a lot better now that I've fixed those problems.

The first actual G-code I ran was to make some shapes out of a piece of aluminium:

You can see there are some ridges in it. These are partly caused by loose Z-axis gibs, and partly caused by the head being out of tram (i.e. not perpendicular to the table in every direction). I have since tightened the gibs but still need to work on tramming.

For my next trick I thought it might be fun to create a toolmaker's clamp, like this:

I didn't have a suitable piece of square steel, so I used round bar instead. This is not a bad thing as it gives me good practice at using the machine to create the shapes I desire. I made this:

The surface finish is OK, and I'm really pleased that I can now machine steel parts. But there are a number of notable problems here. There is still a step in the top surface from the machine being out of tram. The finish on the sides is quite rough because my finishing cut chattered quite badly due (I think) to backlash in the X and Y axes. There were a few false starts while I was getting my toolpath sorted out which led to a few chunks taken out of the end:

But mainly I am foolish and drilled the holes in the wrong axis! So this is now completely useless. Oh well. I'll have another go once I've sorted out the backlash and got the machine back together.

The holes are also not centred properly. This is because I had the steel sitting on parallels in the vice, and I wanted to avoid drilling into the parallels, so I drilled the holes by hand. This seems like a problem I'm likely to encounter a lot, so I need to come up with a solution eventually. Maybe I'd just drill the holes most of the way through on the milling machine, and then finish them up by hand?

Halfway through making this part my nice new sharp end mills arrived, and they are a big improvement. I can see on the spindle load meter that the load for the same cut is much lower, which also implies I can take heavier cuts, and the surface finish is visibly better on the parts that were cut with the new end mill.

Backlash

I found that the Y axis backlash was caused by the screws holding the ballnut to the carriage being loose. They are only phillips head screws which I can't tighten satisfactorily, so I have replaced them with allen bolts now. I had to order the bolts, but the delivery crossed the Jubilee weekend so the wait was annoyingly long. It was tempting to reinstall it with the original screws, but I know I'd have to take it apart again and getting it this far apart is a lot of work so I'd rather not have to do it again. But anyway I think that problem should be solved now.

There was also a small amount of X axis backlash. I found that there were plastic spacers in between the ballnut and the holes that it seemingly bolts on to. Originally I was just going to throw the plastic spacers away, but when I came to reinstall the ballnut I found that the holes don't line up. I also found that the screws holding the ballnut on were M5 while the holes are only M4! How did this work??

Some head-scratching later, I realised that the bolts were only threading into the plastic "spacers" and pushing against the surface with the M4 holes, to push the ballnut against an opposing surface, and this is how the ballnut was attached. What a bodge! This is what happens if you buy unfinished projects on eBay.

Ideally I would drill new holes near the M4 holes and tap them to M5, but unfortunately the required hole location overlaps the existing holes so it would be very difficult to drill and tap. I also looked at widening the holes in the ballnut so that I could get M4 screws to line up with the M4 holes, but the ballnut seems to be made of incredibly hard steel and it blunted both of the drill bits that I tried to use.

So my current plan is to replace the 2 plastic spacers with an aluminium one that spans both holes. With an aluminium spacer I will be able to do it up tighter and hopefully it won't come loose. Again these screws had phillips heads so I want to replace them with allen bolts, but I didn't have any long enough in M5 so I'm again stuck on waiting for delivery of bolts.

Flood coolant

I have made a bit of progress on getting flood coolant set up. The main impetus for this is that I was previously manually dripping oil onto the steel, and this makes very oily and sticky chips that are difficult to clean up.

I bought "Sabre Sol 2B" on eBay. I chose this coolant because it is available in 1 litre bottles. Most water soluble coolants seem to come in enormous quantities. The instructions say to dilute it with water somewhere from 5% to 10% depending on whether you're milling or drilling. Since I'm doing both I just guessed at a concentration somewhere in between.

My main concerns with the coolant are whether it will rust the precision surfaces on the machine, and whether it will short out any electronics it comes into contact with. I stuck my multimeter probes in it (on the 2 Megaohm setting) and didn't manage to measure any conductivity at all unless the probes were touching each other. So I guess it's non-conductive enough? I will not deliberately submerge my connectors in it, but I think getting it splashed on them won't be a problem.

I had some drops of coolant sitting on the machine overnight and they seem to have evaporated without leaving any rust behind, so that's looking like a winner too. I expect it's designed not to cause rust, but it feels very odd to deliberately spray 90% water onto steel surfaces that you care about.

Here's a video showing my coolant system as it stands at the moment:

The big problem is that it pumps much faster than it drains, so if it were running continuously it would flood the tub and run the pump dry. So I need a bigger drain hole.

The pump I got was this one on Amazon. The Amazon listing says it can pump 2000 litres per hour and lift water 15 feet, but the markings on the pump only say 1.5 metres. Either way it seems more than adequate for my use case.

Fly cutter

Once I've successfully made a toolmaker's clamp, I think my next milling project will be a fly cutter.

This is basically like a lathe tool, except it mounts in the spindle of the milling machine and spins around. The cutting point scribes a much larger radius than a normal end mill, and it is supposed to give a superior surface finish compared to machining lots of parallel lines with a normal end mill.

The first thing I would do with the fly cutter is re-cut the top surface of my vice jaws to fix a previous blunder...

Obviously, 5 years of experience in FreeCAD compared to about 3 hours in SolveSpace is not a fair comparison. Also: I love FreeCAD. Like... a lot (call it Stockholm Syndrome if you want). Please bear that bias in mind while reading. I also think 3 hours is just enough time to run into loads of things that I can't do in SolveSpace, but not enough time to discover great new things that I couldn't do in FreeCAD. Most of this post is from the perspective of "what's different in SolveSpace compared to FreeCAD?". If you're a happy SolveSpace user and you think I got something wrong, please correct me!

Whether you're using FreeCAD or SolveSpace, you should definitely run an up-to-date version. The version available in apt is very out of date in both cases. For FreeCAD I recommend the latest weekly build AppImage (not the latest stable on the website, that one is also woefully out of date). For SolveSpace I recommend the solvespace snap package.

Overall workflow

I mostly use the "Part Design" workflow in FreeCAD (distinct from "Part", just to make things easy to understand). The workflow in SolveSpace is almost exactly the same as Part Design, so I found it very easy to get started, after watching a 9-minute video of operating the user interface. Most of it drops straight into slots in my brain that already exist for Part Design.

The black background colour for the UI in SolveSpace is an unusual choice.

It makes the software feel much more foreign and confusing than it really is. But it also contributes to the general "feel" of the program. You feel like you're working more closely with the geometry of the part than you do in FreeCAD. So I don't know if the weird colour scheme is a good thing or a bad thing overall.

What's better in SolveSpace?

The SolveSpace workflow is more cohesive in a way that is hard to give a feel for if you don't actually try it out.

In FreeCAD you have one view for creating a sketch, and then when you're done with the sketch you close it and you have a separate view for turning the 2D sketch into a 3D feature. In SolveSpace it's all the same view. In particular this means that defining things like the length of an extrusion or the angle of a revolution are done using exactly the same constraint solver as the sketch uses! This was a revelation to me.

In FreeCAD you have this amazing constraint solver available in 2D for your sketches, but then when you want to turn them 3D, it all disappears and you have to choose between a simple length, or a pad up to an existing face, or a pad up to an existing face with a custom offset, and so on. The available options are quite extensive, but not exhaustive, and are wholly unnecessary in SolveSpace. In SolveSpace, the extrusion is just dimensioned using constraints. You can very easily do complicated things to decide the length of an extrusion without doing any maths in your head. Just define construction lines in 3D space and constrain the extrusion to meet your construction lines. Easy peasy. This is very powerful and I wish FreeCAD worked this way.

I also like that the SolveSpace "shown" checkbox can make many "groups" visible at the same time. A "group" in SolveSpace is roughly the same as any single item in the project tree in FreeCAD. If you turn several of them on at once (the default behaviour is to show them all at all times) then you get to see all of the elements in every sketch at once (but you only see the constraints of the active group):

In particular, these can then be used to reference other geometry to, to create new sketches, and so on. There's no need to create datum planes or datum lines like you do in FreeCAD. You can just create construction lines directly in 3D space, referencing any geometry in any "group". I think this is a big part of what makes SolveSpace fun to use.

What's worse in SolveSpace?

Fillet/chamfer

There are no fillet/chamfer tools in SolveSpace. This was a big disappointment. The best way to create a fillet or chamfer is to model it in the 2D sketch. This obviously doesn't work for the edges of the flat faces created by the extrusion. In those cases you need to create a new sketch and cut it away from the edges of the extrusion. You also need to make sure that the edges of your sketch do not exactly line up with the edges of the existing part, otherwise SolveSpace gets confused and forgets to fill in some of the adjacent faces, which leaves holes in your part:

The solution is to space the bounds of the fillet away from the existing part geometry:

This whole method is annoying and time-consuming and will result in SolveSpace-modelled parts having sharper edges than FreeCAD-modelled parts with almost 100% probability, because softening the edges is too much trouble.

I'm not saying FreeCAD fillets and chamfers are bug-free! Far from it. Few among us is the FreeCAD user who has not lost an hour's work when the fillet segfaulted an unsaved project. But with time you do learn what kinds of things are likely to crash the filleting tool, and mainly you learn to save before applying risky fillets.

Thickness

I don't use the Part Design Thickness operation very often, but when I do it is a big time-saver. It allows you to model things like an electronics project enclosure as just the outer shape, and then turn it into a hollow part with uniform thickness and 1 open face with a single click. In SolveSpace I think you'd have to manually model both the external and internal geometry, I'm not aware of any way to automatically create the uniformly-sized offset.

Rotation

I can't figure out how the 3D rotation works in SolveSpace. There are 2 options (the default one and "turntable") and they are both equally baffling to me. They both have the peculiar property that you can drag the mouse around in a circle while rotating, and when the mouse gets back to the start point on the screen, the part will not have returned to the starting orientation. I expect this is the sort of thing that you get used to with time, but for now rotating the part is very much a guessing game for me. It doesn't work like any 3D mouse rotation I am familiar with.

Pad direction

Normally when you create a Sketch and make a Pad (in FreeCAD) or a Union extrusion (in SolveSpace), it comes "up" out of the sketch, and towards the direction you were looking at it from, and when you make a Pocket (FreeCAD) or Difference extrusion (SolveSpace) it goes in the other direction. All very good and sensible.

Update: u/henrebotha points out that you can reverse an extrusion just by clicking on the surface and dragging it in the opposite direction. This is great! It's actually quite intuitive, I don't know why I didn't think to try it. I must have still been stuck in the FreeCAD mindset of trying to interact with the GUI rather than the part.

Sketches

What's better in SolveSpace?

SolveSpace visually shades the enclosed area of the sketch so that you can see which parts are "positive" and which parts are "negative". This is good and FreeCAD should copy it.

(It's a dark blue-ish colour on a black background; for some reason it's easier to see in the application than in the screenshot)

SolveSpace also helpfully points out where your sketch is non-closed so that you have half a chance of fixing it.

It's very annoying to spend a long time on a complicated FreeCAD sketch, and then go to create a Pad, only to find that it can't create the Pad because the sketch is not a closed shape. You then have to manually check every vertex in the sketch to find out which ones aren't connected properly, only to find that 2 points that looked the same are actually 0.01mm apart.

Update: u/strange_bike_guy points out that FreeCAD has a "validate sketch" tool that can automatically find missing coincidences within a given tolerance.

Once you've created a sketch in 3D, I love that you can click and drag the unconstrained face and move it around in 3D space. I grant that this is usually not particularly useful, because you'll usually want to add a constraint to the length anyway, but it adds to SolveSpace's feeling of a direct connection to the part geometry. In FreeCAD you just have a text box to type a length into and absolutely no chance of manually dragging the extrusion around.

In FreeCAD if you want a sketch to reference some geometry that already exists on the part, you need to use the "external geometry" tool to bring it into the sketch. This never really occurred to me as an unusual thing to have to do, but in SolveSpace that tool just... doesn't exist. You just don't have to do it. All of the geometry is already available to use, you don't have to do anything weird to bring it in. This just seems obviously better. I'm not sure why FreeCAD isn't this way.

What's worse in SolveSpace?

Sketch attachment

In FreeCAD you can create a sketch on any existing flat face of the part very easily. Just select the face and then click the "create sketch" button. In SolveSpace, even though flat faces are highlighted in a weird yellow mesh when you hover over them, selecting a face and then trying to create a "workplane" based on it does not work.

Even though if you select a single point, creating a workplane based on the point works fine. Doesn't a face describe a plane better than a point does? I don't really understand the thinking there. What you have to do instead is select 2 of the lines that form the edges of the face, then a point somewhere on the face, and then create the workplane. It works fine but feels over-complicated in the common case of creating a sketch coplanar with an existing face. I also don't see what the point adds over the 2 lines. Don't 2 coplanar lines describe a plane on their own? It's also difficult to make a workplane on a circular face, because you don't have any straight lines to use.

In FreeCAD once you've created a sketch on a face, you still have the option to adjust the attachment position of the sketch so that it is offset from the face you attached it to. I don't do this particularly frequently, but when I do it is a big time-saver compared to setting up a datum plane.

In SolveSpace, you have to manually set up some construction geometry that is offset from the face and then create your workplane on your construction geometry. It works, and it's not that much more work than just creating a workplane off existing geometry. But what is much worse in SolveSpace is that once you've created the workplane, if you didn't have the foresight to add an offset in the first place, you can't add an offset at all. If you change your mind about where you want the sketch, you need to delete it and start all over again. (And, actually, I'm not sure how this works if you change your mind about where you want a sketch that is several steps back in the Property Browser; do you need to remake everything that came later? I hope not. Probably someone who knows more about SolveSpace than me would know a way to adjust a sketch attachment without breaking everything that came later).

Multiple constraints

In FreeCAD you can select any number of lines and set an equality constraint on their lengths. Or you can select any number of points and put a vertical constraint on them to make them all lie in a vertical line. In SolveSpace, these constraints can only operate on 2 lines or 2 points respectively. This means when you have a lot of lines to make equal, or a lot of points to make vertical, it takes twice as many clicks in SolveSpace as it does in FreeCAD.

Redundant constraints

In FreeCAD there is an option for "auto remove redundants".

This allows you, for example, to set points coincident with each other, and if the points already have some constraint that meant they were already coincident in one axis, that constraint can be automatically removed from one of the points to avoid over-constraining the sketch. SolveSpace doesn't appear to have any workaround for this. The closest thing it has is "ignore redundant constraints", but see the "Bugs" section below for why this isn't satisfactory. The best way to avoid this is to try to remember to constrain points to lines rather than to other points. And when you know you're going to make 2 circles symmetrical about the origin, for example, remember to place one circle on the X axis, and remember not to place the other circle on the X axis, otherwise after you add a symmetry constraint across the Y axis, you'll have 2 separate constraints putting the circles on the X axis, which makes the sketch over-constrained. Just a bit more forethought required in SolveSpace than in FreeCAD.

Unconstrained elements

In FreeCAD, unconstrained elements in sketches are shown in a different colour to constrained elements (white instead of light green).

This makes it relatively easy to locate the unconstrained parts. SolveSpace doesn't have any obvious way to point you towards the unconstrained elements, you just have to think about it and find them on your own.

Symmetry across a point

SolveSpace can't do a symmetry constraint across a point. It can only make 2 points symmetrical across a line. Making 2 points symmetrical across a point is the fastest way to centre a rectangle on the origin, and maybe 50% of my parts in FreeCAD start this way. In SolveSpace you'd have to make the top edge symmetrical across the Y axis, and the left edge symmetrical across the X axis. (And then you'd have to manually remove the vertical and horizontal constraints on the rectangle, else it's over-constrainted). It just takes more clicks in SolveSpace.

Cross-section view

In FreeCAD's Sketcher workbench there is a button called "View Section" that turns the view into a cross-section of the part along the sketch plane.

That means every feature on the part that would sit between the "camera" and the sketch plane is hidden, so that the sketch is unobscured. This is a very useful feature. I couldn't find anything equivalent in SolveSpace, which means if you work on a sketch that intersects the existing solid, you're going to struggle. The closest thing I could find was "draw occluded lines" so that you can see the lines on your sketch even though they're not on top. But this can presumably get very cluttered in a complicated part.

Assembly

FreeCAD has a lot of different assembly-related workbenches. I don't use any of them. I think there are too many options and they all work too differently, and there is a sort of paradox of choice where I can't work out what I should be using, and I work around it by manually positioning and rotating separate bodies using the property editor instead.

SolveSpace seems to have a simple assembly system built-in. I have not used it yet, but it seems so much simpler that if I do use SolveSpace for a non-trivial project then I will almost certainly use the assembly system.

CAM

CAM seems to be missing from SolveSpace. I expect even if you really liked SolveSpace and hated FreeCAD, the best way to do CAM would be to model your part in SolveSpace, and then load it in FreeCAD to use the Path workbench. (Unless you want to do 3D CAM, in which case you should definitely try Meshmill instead!).

There are some CAM-related options in the "configuration" window (window? tab? zone?), namely "cutter radius offset" and "exported g code parameters" but I couldn't find anything in the user interface that would actually generate any G-code. I'm not sure if it is not implemented yet or just too hard for me to find. But I didn't look very hard, and besides, even if it is there, the fact that "cutter radius offset" is a global option does not suggest very advanced toolpath generation.

Bugs

FreeCAD bugs

FreeCAD has plenty of bugs of its own. In FreeCAD, bugs frequently segfault the application. If you try to do anything complicated in FreeCAD, it is going to segfault on you. That's just part of the experience. Remember to save frequently. When it segfaults, just reload FreeCAD and try the same thing again. Most of the time it will work. If you have found something that reliably segfaults FreeCAD then you just need to find a different way to do what you're trying to do. But segfaults are just part of life when you use FreeCAD. The first time it happens it feels like a complete impediment to getting any work done, but eventually you realise it's just a low percentage drain on productivity and not a big deal. I'd rather have a Russian Roulette filleting tool than no filleting tool at all! And I would definitely say stability improves with each new release I download.

FreeCAD has an automatic project recovery tool that is meant to restore your unsaved work after a crash. I'm pretty sure it doesn't work. It's some sort of complicated cruel joke. The only time it ever restores what you were working on to the state you wanted is when it was a test part that you didn't need. But it doesn't matter. Just learn to save your work!

I didn't have a single segfault from SolveSpace, which is good. In fact I didn't find any bugs that would trash your work without warning.

Property Browser window going all black

I found that the Property Browser window sometimes goes completely black and I need to waggle the mouse over it to get it to redraw. I don't know why.

Unwanted open faces

I consider the thing where it makes open faces if you try to make a fillet that is exactly coincident with existing geometry to be a bug. But let's say you don't! Let's say you want open faces when you try to make a fillet coincident with existing geometry. Even then, surely this is a bug:

I tried to create a cylinder that partly passed through open space and partly joined to an existing solid, and it just didn't work at all. It left some open faces for no reason that I can discern. I have come across this sort of thing in FreeCAD, but only a handful of times ever, and always when doing something very weird. In SolveSpace it seems to happen a lot. I expect this will get better in time.

"Ignore redundant constraints"

If you enable "ignore redundant constraints" on a sketch, then SolveSpace always thinks that sketch is completely constrained. This is a real shame because I really like the way that SolveSpace shows the number of degrees of freedom of each "group" in the Property Browser. In FreeCAD, if you want to know the degrees of freedom of a sketch, you have to open up the sketch to find out. If you want to check whether all sketches are fully-constrained, you need to open every sketch separately to check. In SolveSpace you can just eyeball the list of groups for any that don't say "ok" in the "dof" column.

"Draw occluded lines"

There's a button called "draw occluded lines" which seems to have 3 modes. In one mode, occluded geometry is not shown at all. In another mode, occluded lines are shown but occluded points are hidden. In the final mode, occluded lines and points are both shown. When you click the button, it seems to switch from the currently active mode to one of the other 2 modes chosen at random. If there is logic to the choice, I was not able to fathom it. I'm not sure whether this is a bug or whether the logic is too complicated for me to understand. It's easy enough to work around though: just keep clicking the button until you get what you want.

Verdict

SolveSpace is fun to use in a hard-to-describe way that FreeCAD is not. It is also much more stable (i.e. it doesn't segfault). It is also much more immature than FreeCAD, and much less feature-complete. Overall I think FreeCAD is a more productive CAD program, mainly just because it can do fillets and chamfers, and you encounter open faces much less frequently, but also because of the "long tail" of little things that can be done in fewer clicks in FreeCAD.

That said, I found SolveSpace much faster to become productive in than FreeCAD was. Part of this is probably because I have already spent the time to learn the Part Design workflow in FreeCAD, and SolveSpace can piggyback on my learning investment. But I do think SolveSpace is genuinely easier to learn than FreeCAD. There is just much less there in SolveSpace, so tentatively exploring the UI by clicking on things and seeing what happens is actually a viable strategy in SolveSpace. Not so much in FreeCAD. FreeCAD seems to have a lot more things available to click on, without a correspondingly large number of extra features.

If you just want to create parts for 3D printing and you haven't used either SolveSpace or FreeCAD before, start with SolveSpace. If you want to make parts for CNC, probably start with FreeCAD because you'll probably need FreeCAD anyway when you want to make toolpaths. If you use FreeCAD for productivity and you want something different to use for enjoyment, and on small projects, SolveSpace is worth your time. If you are competent in SolveSpace and you want something more powerful, FreeCAD is worth your time.

Potential easy improvements to SolveSpace

(I don't know if these things are actually easy, but they feel "close to the surface").

It would be cool if hovering over a "group" in the Property Browser highlighted all of the geometry that belongs to that group. In the event that this is ambiguous: let's say it highlights all of the geometry that would disappear if that group were made invisible. I often have trouble in FreeCAD working out which sketch created a particular feature, if I need to go back and change it. If I'm trying hard then I rename the features that I create so that I can tell what they are, but most of the time I don't bother. Manually toggling visibility of Pad001, Pad002, Pocket001, Pad003, etc., until I work out which one my feature came from is very time-consuming. It seems like it would be quite easy for SolveSpace to be able to do this with just a hover in the Property Browser.

Whenever the view needs to reorient, the rotation is animated. The animation is infuriatingly slow. It's probably only a second or so, but it feels bad. It should be more like 100ms in my opinion. Most of the time SolveSpace is a perfect servant of the user, always ready to process your inputs as fast as you can provide them. But that is broken for a little while whenever it rotates into a new position. I looked for an option to make the animation faster, but I couldn't find one. Animated rotations are useful because they help you stay mentally oriented with the part, but I think I would rather have no animated rotations at all than animations that are this slow. FreeCAD also has animated rotations but they are optional and you can change how long they take in the settings. SolveSpace should just make this configurable. I mostly ran into this problem after I accidentally rotated a part while editing a sketch, and then hit "W" to put the view back to the workplane, but then I had to wait for the animation to be done even though I already knew exactly how the workplane was oriented to the part.

It's a "mini mill". I don't know what company in China actually makes these, but they're a small manual milling machine sold under various brand names. They look like this:

The man I bought it from claims to have spent £1000 on just the machine. There are 6 motors and drivers (3 axes, plus 2 rotary axes not yet built, plus a spare), which come to £600 on AliExpress just by themselves, and the purchase also included the pictured hotel-laundry-basket enclosure, a very sturdy (and heavy) welded steel stand, a PC, a touchscreen for the PC, and other assorted stuff that all adds up (2x DC power supplies, parallel port breakout board, electrical cabinet, ER32 collet set, ballscrews, belts, pulleys). Not to mention the time and materials that went into making all the parts for mounting the motors. I think £500 for the lot is a steal!

However I did have to drive to Bingley to collect it, which is a 10-hour round trip, so you can add on another £100 just in fuel.

I've been spending a lot of time working on Meshmill recently so haven't done much with the milling machine project, but today I got all 3 axes moving for the first time.

LinuxCNC

The previous owner had set the control box up with a parallel port break-out board, and supplied a suitable PC for Mach 3 to control it over a parallel port. I'm not interested in Mach 3 because I like freedom, so I thought the easy route would be to use LinuxCNC to drive the parallel port, rather than install Grbl on an Arduino like on my router.

So far I have had quite a lot of trouble with LinuxCNC, to the point that I am surprised anyone bothers with it. It is a lot more complicated than Grbl, for almost no benefit. I grant that some of the complication makes it more flexible, like the hardware abstraction layer, and all the different options for GUIs, and the ability to swap in different kinematics modules for different types of machine.

But a lot of the complexity is a result of trying to use PCs to do real-time electronics. For example, you have to make sure the parallel port is set to the correct mode in the BIOS configuration, otherwise it only works in one direction. You need to run a latency test to find out how much jitter can be expected from process scheduling so that you can set the main loop frequency correctly. The main loop frequency sets your maximum step rate, which is shockingly low compared to Grbl on an Arduino.

And it turns out that there is some problem with the GPU in the PC that I am using, so I have had to disable the glx kernel module otherwise the computer locks up every few minutes. It also turns out that most of the LinuxCNC GUIs require OpenGL, so most of them don't work. I'm using Qtaxis currently, which works fine but is strictly less convenient than UGS Platform, which is what I use with Grbl (most notably, jogging the axes takes more clicks, and manually setting the current position requires typing G92 commands at the console instead of editing the number in the DRO). I'll probably make my own Qtvcp GUI if I don't come up with something better.

But for now I'm persisting with LinuxCNC. Maybe I'll see the light eventually.

Servo drives

Rather than traditional stepper motors, this machine came with Leadshine "hybrid servo motors". They are controlled just like stepper motors (you give step & direction signals to the driver), but I gather that the motor internally works like some sort of cross between a true servo motor and a stepper motor with an encoder on it. I don't completely understand how it differs from a true servo motor, but they're variously called "hybrid servos" and "easy servos" rather than just "servos" so I'm guessing there is some important difference.

When you crash a machine with normal stepper motors, the driver has no way to know, so the motor loses steps and the job carries on and the work piece gets destroyed and there is no way for the control software to do anything about it.

With these Leadshine motors, the driver continuously monitors the position of the motor and if it differs from the commanded input, it tries to correct it. If it is unsuccessful, and the following error gets too large, then the driver cuts power to the motor and goes into an alarm state. There is an alarm output from the driver which can be connected to the control software so that you can know something bad has happened, stop moving the other axes, and maybe not cause as much damage.

So far so good.

But once the driver has got into the alarm state, I'm not actually sure if there is any way to get it out of the alarm state short of switching it off and on again. I tried toggling the enable pin but it seems to do nothing once the alarm state is active. I could add a relay to the power input so that LinuxCNC has a way to power cycle the drivers, but that seems like a hassle. I'll probably just manually switch the power off and on again if that's the only way to clear the alarm. I don't expect to have to do it very frequently anyway.

Spindle motor

The spindle motor is (I think) 500 W or 750 W. This seems very low. My router has a 1500 W spindle motor, and the router is a lot less rigid than the mini mill, so I would expect the mini mill should be able to put much more power into a cut than the router can.

Another one of my weird eBay "bargains" recently was a 2200 W water-cooled spindle motor that leaks. I think it would be straightforward enough to fix the leak and fit it to the mini mill, but I'm not sure whether it would be a good idea. The 2200 W motor spins up to 24000 rpm, which is fine for a router but way too much for more traditional milling. It's possible that reducing the speed of the 2200 W motor from 24000 rpm to something more suitable would put it at a point on the torque curve where it is producing less power than the original spindle motor. So I'm not sure exactly what to do about that. For now the path of least resistance is just to wire up the motor that already fits, rather than make a mount for the other one.

When I got the machine, the spindle motor was not fitted, which confused me because everything else about the mechanical assembly seemed complete. I have now worked out why the motor was not fitted.

The mounting plate for the spindle motor is a beautifully-machined part, but it all lines up so that you can have any 2 of:

• belt installed
• motor bolted to mounting plate
• mounting plate bolted to machine

Getting all 3 at once seems impossible. I'm going to keep working at it though. There are lots of possibilities for the order of operations (motor on plate first, or plate on machine first? belt on before or after plate?), and maybe if I switch to hex head bolts instead of allen bolts I'll have more clearance to get the spanner in. Maybe this puzzle has a solution after all! Especially if I allow myself to skip out a couple of the less-important-looking bolts.

Emergency stop

The emergency stop switch provides a signal to the control software (boooo! The emergency stop should always cut power to the machine, not speak to the software). I set it up using the Stepconf tool in LinuxCNC, but found that it wasn't working at all. LinuxCNC didn't recognise that the switch was being pressed. (Which is precisely why the emergency stop needs to cut power on its own, not speak to software).

I eventually found out that the parallel port breakout board I'm using requires the 12-24V power supply to be present as well as the 5V supply if you want inputs to work.

I had assumed that the 12-24V supply was only there to get 10V for the spindle speed output, but apparently not. It is even more confusing because the outputs work just fine with only the 5V supply connected. So if you're having trouble getting LinuxCNC to read inputs from the parallel port breakout board: make sure you have the 12-24V supply connected as well as the 5V one.

Limit switches

The machine doesn't have any limit switches installed. The eBay listing showed proximity sensors in one of the pictures, presumably intended for use as limit switches, but when I got everything home I found that these had not been packed in any of the boxes.

I don't think it's a big deal to be without limit switches. I don't have any limit switches on my router and that seems fine. You need to set the origin of the coordinate system before you start a job whether you have limit switches or not. In the absence of an automatic tool changer or tool height setter, I don't think there's any advantage to having the concept of a global coordinate system, so I probably won't bother fitting any limit siwtches.

However LinuxCNC does seem to be more of a stickler for global coordinates than Grbl is. If you try to drive the machine outside the software-configured bounding box then it causes an error instead of just doing what you asked. Probably the solution is to set the bounding box really large so that LinuxCNC always thinks it's inside it.

Enclosure

The enclosure is made out of a plastic hotel laundry basket. It seems like a really good idea. It's a lot less work than fabricating a custom metal enclosure, easier to keep watertight, and surprisingly sturdy. The door/window is a plastic panel that drops in from the top and slides inside some rails. It's a bit of a wiggle to fit the panel between my garage ceiling and the enclosure, to put it on or take it off. I don't know if this will prove to be annoying. I may put it on hinges if it is.

Coolant

Once I have the machine properly working and installed in its enclosure, I think I might like to add a flood coolant setup. I would do this with an aquarium pump. I would probably want to make sure the enclosure is slightly out-of-level so that all the liquid pools up in one corner, and then drill a hole in that corner to let the coolant drip back into the reservoir for recirculation.

If you run Linux (on x86_64) and want to try it out, you can download the AppImage from the Github repository.

Download the latest Meshmill AppImage

If you try it out (successfully or otherwise!) please let me know how you get on, what you struggled with, etc. And I'd love to see pictures of your parts :).

I have filmed an introductory video showing how to use it to make a part:

What is 3D CAM?

You can machine a lot of parts by machining features at different heights in flat planes. This is called "2.5D" machining, because although the produced parts are 3-dimensional, all of the tool movements are only in the XY plane, with the Z axis only moving to get into position before the start of each level. You can only make surfaces that are horizontal or vertical.

(From Welsoft)

Most of the parts I make on my CNC machine are made with 2.5D toolpaths generated by FreeCAD.

To make parts that have angled or curved surfaces, you need to move the tool in the Z axis at the same time as the X or Y axis. This is "3D" machining, and this is what Meshmill is designed for.

(From Welsoft)

FreeCAD has some buttons that are supposed to be able to generate 3D toolpaths, but I've never been able to get them to work. PyCAM makes 3D toolpaths but I always ran into Python errors when trying to run it. BlenderCAM is supposed to be good, but I only found out it exists after I started working on Meshmill, and although I have used Blender on occasion, I find it too complicated.

How does it work?

In 2020 I wrote a tool called Pngcam that takes a heightmap as input and produces G-code as output. It also has another tool that takes an STL model as input and produces a heightmap as output, so that they can be used together to make G-code from an STL model. Pngcam is a 3D CAM program, but it is inconvenient to use because it is operated from the command line.

Meshmill is an Electron application providing a GUI frontend to Pngcam.

Since getting the GUI working, I have also finally got around to adding more advanced features to Pngcam, such as ramping on plunge cuts to reduce tool load, skipping out cuts on the flat top surface of the material, reordering cutting segments to reduce time wasted on travel moves, and producing an output heightmap showing what the material will look like after one operation so that it can be used as input for the next operation to stop wasting time machining air.

I've also reimplemented Pngcam in Go, partly for performance reasons, and partly because Go binaries are easier to package in an AppImage than the Perl scripts were.

(The problem I had with the Perl scripts was that they used GD to handle images, and GD is an XS module that uses libgd. So I dutifully packaged libgd.so.3, and GD.so, and wrote a wrapper script that mucked about with LD_LIBRARY_PATH to get it to work, and I succeeded. Great success! But then I tried to launch the program on a different computer, with a different Perl version, and it didn't work, because XS modules are tied to the Perl version they were built against. So then I worked out how to package a matching Perl installation in the AppImage as well, and I expect that would have been enough, but it seemed very distasteful, would be very annoying to update, and added about 20 megabytes to the size of the AppImage.)

What's good about it?

Meshmill's primary feature is that it is (intended to be!) easy to use. Importantly, this includes "easy to install": there is no compiling and no dependencies, you just download the AppImage and run it.

I have found other options for open source 3D toolpath generation very difficult to get started with, to the point that I have never succeeded in generating a 3D toolpath except with Pngcam and Meshmill. If you are already using a different open source 3D CAM program: chances are it is better than Meshmill and you should stick with it. If you haven't had success with other open source 3D CAM, you could try out Meshmill.

What's bad about it?

Despite recent improvements, Pngcam's planning is suboptimal. It still spends more time on travel moves than it could do if it ordered the cutting segments more intelligently. (Ordering cutting segments optimally requires a Travelling Salesman Solution, but I'm sure it could reasonably do better than it currently does).

It can generate travel moves that touch the surface of the part, which in combination with the "omit top" setting, can result in taking small gouges out of the top surface of the part, if the Z axis is zeroed slightly below the top surface. See my sake set picture below for an example of this.

There are still some very useful features missing from the GUI, primarily:

• some way to generate hold-down tags
• a way to rotate the part to more faces than just top and bottom (probably I'd aim to duplicate the PrusaSlicer UX here)
• a way to start from a heightmap instead of an STL model
• a way to export the "stock" heightmap from one job and import it into another

My sake set

The first part I have made using Meshmill is this:

It is an insert for a box to hold my sake set (a jug and 4 little drinking cups).

Wanting to make this part is what finally pushed me to make the GUI for Pngcam in the first place, so I'm glad I've now succeeded in that sense. Hopefully that doesn't mean I've got what I wanted out of Meshmill and will entirely stop working on it now!

I'm going to coat this part in flocking to give it a fabric-ish texture, and build a box with a lid around the outside.

The short answer is that it works a lot better than I expected, and there is a fun trick to creating toolpaths with normal 3-axis CAM software. If you are tempted to build one of these, you should do it. The only expensive part is the chuck. You can either borrow the chuck from a nearby mini lathe, or failing that you can screw workpieces directly onto the plywood gear, or 3d print some sort of fixture. You'll want to keep the cutting forces low anyway, so the workholding doesn't need to be anything clever.

Principles

There are a few key things you need to know in order to do the calculations to control the rotary axis. The first one is the pitch circle of the gear. For metric gears, the diameter of the pitch circle is defined to be the tooth module multiplied by the number of teeth. The circumference of the pitch circle is the diameter multipled by pi. If you move the Y axis through a distance equal to the circumference C of the pitch circle then your gear will perform 1 complete revolution.

Distance for 1 revolution, C = π * module * no. of teeth

In my case, the tooth module is 3 mm and there are 30 teeth, so for me C = π*3*30 = ~282.74 mm.

If you want to move through an angle of α°, then you can scale the circumference accordingly. For example, to move through 90°, you need to move through a quarter of a revolution, because 90 is a quarter of 360.

Distance for rotation of α° = C * α / 360

Sometimes it's more useful to work with distances than angles. For example, if your part has a circumference of C_part and you want to cut a line of 5 mm on it, how far do you need to move the Y axis? Just multiply 5 by the ratio of the rotary axis's pitch circle circumference to the part's circumference:

Distance on the Y axis = distance on the part * C / C_part

Let's define the ratio R = C / C_part. So if our part is 20 mm in diameter, then it has a circumference of 20π = ~62.83 mm, so R = 282.74 / 62.83 = ~4.50.

So to cut a 5 mm line around its circumference we need to move the Y axis by 5*R = 5*4.50 = ~22.5 mm.

Feed rates need to be scaled by the same ratio as distances, because we care about the feed rate at the surface of the work piece, not at the surface of the table. If we want to feed the tool into our 20 mm part at 1000 mm/min, then we need to command the machine to move the Y axis at 1000*R = 1000*4.50 = 4500 mm/min.

Calculating the feed rate gets more complicated for diagonal moves, because distances and feed rates in the X axis are unaffected by the rotation. You can calculate feed rates for diagonal moves by working out the ratio of the length of the movement on the table to the length of the actual cut on the work piece (using Pythagoras to work out the distances - form a triangle with X distance on one side and Y distance on the other, and the hypotenuse is the total distance of the movement). Multiply your desired feed rate by this ratio. (But if you find yourself having to do this calculation more than once then you should probably try to find a better way to make your toolpaths).

3-axis CAM

There is a fun trick for making toolpaths for the rotary axis:

1. model the part as a flat "unwrapped" equivalent
2. generate the toolpaths to cut out the flat version
3. reconfigure Grbl's Y axis using the R ratio calculated above
4. the toolpath is magically wrapped up onto your cylindrical part

I modelled this ring in FreeCAD as a flat part:

I want the outer diameter of the ring to be 26 mm, so the outer circumference is π * 26 = ~81.68 mm. I modelled the ring as a flat part that is 81.68 mm long in the Y axis, and generated suitable toolpaths to cut the ring out of a flat piece on the table.

For C_part = 81.68 mm, on my machine, R = 282.74 / 81.68 = ~3.46. Then we just need to update the Y axis settings as follows:

Setting	Linear	Rotary
$3 (axis invert)	7	5
$101 (Y steps/mm)	320	320 * 3.46 = 1107.2
$111 (Y max. velocity, mm/min)	4000	4000 / 3.46 = ~1150
$121 (Y max. acceleration, mm/sec2)	50	50 / 3.46 = ~13

The most important setting is $101. This should be set precisely otherwise your toolpaths won't line up properly with rotations of the chuck. $111 and $121 only need to be set approximately. These are limits to how fast the machine can move around, so they need to be divided by R to cancel out the multiplication in $101, otherwise Grbl will try to move the machine too fast, because what Grbl thinks is 100 mm (i.e. what is 100 mm on the work piece) is actualy 346 mm in physical space.

And don't forget to toggle the Y bit in $3, because in rotary mode the top of the part is effectively moving in the opposite direction relative to the tool.

Also don't forget to put these numbers back when you're done otherwise all your non-rotary parts will be mirrored and stretched in the Y axis!

There are a few really good reasons to do the rotary axis CAM this way:

• a lot more tooling is available for flat 3-axis CAM than rotary CAM.
• it will never try to do more than 1 complete rotation, so you'll never drive past the end of the toothed rack, which is a limitation that most rotary CAM isn't likely to be aware of.
• feed rates are scaled automatically, so you don't need to worry about setting your feed rates correctly for X vs Y movements, and in particular you don't need to do any calculations for diagonal moves, and you even get correct arc movements for free.

"Tailstock" support

I 3d-printed a wheel to support the loose end of a long aluminium cylinder. It is all held tight by a threaded bar that runs through the middle of the work piece, chuck, and bearings, secured with a nut at both ends.

This is OK but not amazing. It stops the part from being ripped out of the chuck in plunge cuts, so it does its job in that sense. But it does nothing at all in the horizontal plane, so you still get more flex at the end furthest from the chuck. Also, the wheel is slightly larger than the pitch circle of the rotary axis, which means it wants to roll further for the same angle of rotation. This means the wheel is constantly slipping against the table.

To stop the wheel from slipping we could make the wheel the same size as the pitch circle of the gear, and put a raised "road" on the table for the wheel to roll on. A further improvement would be to replace the wheel and road with another gear and rack, and this would get us some lateral stability as well. If I do much more work with long parts I will probably give this a go.

Rings

I made the wooden ring for myself, and put my initials on it. This was made by hot-gluing a "blank" (cut out with the CNC machine in its normal linear configuration) to a 3d-printed mandrel held in the chuck:

The letters and hexagons are cut first, then painted gold, and then the surface is machined down to expose the wood underneath, leaving the gold paint only in the valleys. Here's a video showing the engraving (sorry about the audio, I don't know what happened):

I made this ring to wear at a wedding I attended at the weekend. Since getting married and getting used to wearing a ring every day, I have really got into wearing rings at special occasions. So far 4 people have told me I should try to sell these rings, and 2 people have told me I shouldn't, so to test the water I have made an initial batch of 5.

I made the batch of 5 in largely the same way as the first one, except with a larger mandrel so I can mount 5 blanks at a time, and I didn't engrave my initials on them because most people have different initials. They're available in my etsy shop.

If I were to make more rings I would do them differently. I think these are weaker than they need to be because the grain runs across the ring in the long direction instead of the short direction. They should have the grain running along the direction of the finger, rather than perpendicular to the finger. Also, instead of making each of the ring blanks individually and then measuring their coordinate offsets and machining patterns on them individually, I would start with a single long tube of wood, with the grain running axially. I'd machine the pattern over the entire surface of the tube without regard for where each ring is going to start and finish, and then I'd part off individual rings from this large tube. I think this would be a lot less work.

Cylinder Puzzles

I've designed a puzzle that involves shuffling rods back and forth through a cylinder, to allow the inner "puck" to be removed. It's a type of maze puzzle where you can see all the parts of the maze, but it's still difficult to find the solution because the possible movements are so hard to see.

The one on the left is 3d printed and the one on the right is machined out of aluminium. The machining is not great, and the part is worse at the end that was further from the chuck, because my "tailstock" support wheel doesn't provide a lot of support, but it's a very promising start. I still need to make metal versions of the puck and the sliding rods.

For this part I did not use the 3-axis CAM trick. I'm using a similar ASCII-maze-based workflow for this puzzle as I am for Party Puzzling, so I wrote a Perl script to turn the ASCII art maze into G-code.

I started by testing out the machine and the toolpaths on a random piece of plastic pipe:

It worked, but the (PVC?) plastic pipe does not machine very well at all! It makes a real mess. I eventually worked up the courage to try it on aluminium:

I broke the endmill on the first attempt because I was trying to use feeds and speeds borrowed from the web. It seems that for cutting aluminium with a router you need to use much higher RPM than real machinists would use. I don't know why. From first principles I think there should be an optimal relationship between RPM and feed rate, so that you're taking a proper chip and not rubbing the tool against the work, but it seems that on a router it cuts a lot better (and doesn't break the tool!) if you take much smaller nibbles than what machinists will tell you to use.

I don't know, maybe I just don't understand.

I found that running the spindle at 8000 rpm and feeding at 800 mm/min worked best. It's a 6mm 2-flute carbide end mill.

The Librem 5 does not run Android or iOS, it runs a proper Linux operating system (PureOS, derived from Debian). There are hardware kill switches for the baseband modem, the camera and microphone, and WiFi and bluetooth. The phone's main selling point is that it doesn't spy on you.

If you don't care that your phone is spying on you, then the Librem 5 is not for you. If your response to complaints like "the camera app is hard to use and can't take video" is "why don't you just buy an iPhone?" then you have totally missed the point. The Librem 5 is not in the same category of things as the iPhone. Anybody who would enjoy a Librem 5 would not be able to tolerate an iPhone (and vice versa).

The short version

(For context, my previous phone was a OnePlus One running Ubuntu Touch.)

The Librem 5 is a lot better than I had been led to believe. It's much faster and more responsive than the OnePlus One, and although I had heard a rumour that phone calls don't work, they work perfectly. Great success, good job Purism. If you are the kind of person who dismisses Android and iOS out of hand because you like freedom, then the Librem 5 is a good phone for you and you should buy it (but, again: if you don't care that your phone is spying on you, then the Librem 5 is not for you).

Notable parts that work:

• WiFi works, no setup required beyond selecting a network and entering the password.
• Calling works, no setup required beyond installing a SIM card and rebooting.
• SMS works, no setup required beyond installing a SIM card and rebooting.
• 4G works after you manually setup the APN settings; the phone defaults to Tesco Mobile settings, for some reason.
• Hosting WiFi hotspots works, no setup required beyond enabling the option.
• Email works, just the normal IMAP and SMTP setup.

Notable parts that don't work as well as they should:

• The camera app works, but you have to manually set gain, exposure, white balance, and focus. It also seems to take about 10 seconds to save a JPEG to the disk, so you won't be taking photos very rapidly, and it has no way to record a video, and the quality is bad if there isn't enough light. I find the camera too annoying to use, it is the phone's largest drawback.
• The default browser has a few input bugs: if the text in a search box is longer than the box, and you want to edit some text that has scrolled out of view, you just can't. There is no way to horizontally scroll the text box in the browser. This is not a general UI bug, it works fine in other apps. There is also no way to open a link in a new tab: long press on a link pops up a context menu, but you can't select anything in the context menu because it disappears as soon as you take your finger off the screen. Firefox is also available (sudo apt install firefox-esr) and doesn't have either of these problems.
• The maps app has a way for you to search for start and end destinations. The search is really bad, it doesn't recognise postcodes and it thinks all towns are in America. Once you have managed to correctly select a start point and an end point, and you want to start navigation, you can't, because the UI disappears off the right hand edge of the screen so you can't reach the "start navigation" button that presumably exists. If you rotate the screen into landscape mode then it now disappears off the bottom edge of the screen and you still can't reach any buttons.
• The battery life seems short. I'm pretty sure that I charged it up to 99% when I plugged it in this afternoon. It's now 10pm and I just went to check something in Firefox and found that the battery has died already. And I haven't exactly been using it heavily. It's possible that I misunderstood how much I had charged it, but so far this is a bad sign.

The rest of this post is roughly a stream of consciousness of my thoughts as I first interacted with the phone.

Unboxing

It's big. It's heavy. The outer casing is aluminium, I expected plastic.

It takes USB-C, which is annoying, as I like to live in the past. The charging cable has USB-C on both ends which means it's not possible to charge from my normal USB charging dock, so I have to use the bundled UK-to-USA-to-USB power adapter. The UK-USA plug adapter feels sturdier than some that I've used, but not as sturdy as a real 3-pin plug. The USB-C cable is very chunky, I'm not sure if this is a good sign as it carries lots of power, or is just to fit with the aesthetic of the rest of the phone.

It asked me for a disk encryption passphrase. At first it wasn't clear whether it was asking me to set a passphrase, but no. The default passphrase is "123456", once you type that in it boots up.

It seems fast. It booted faster than I expected. I can't drag down from the top bar to access the WiFi setup. How do I WiFi? Oh, you have to single tap on the top bar, not drag. I need to get used to that. On Ubuntu Touch you have to drag everything.

Backups

Before I got very far a box popped up asking me to configure backups. It's not clear on whether they are local or remote, and who would be hosting them if remote. Normally I would ignore this popup, especially on a phone, but today I'll play along.

Once you've agreed to backup your data, you now get to decide where you want to store it: Google Drive, "network server", or "local folder". What are the chances it supports sshfs?

I tried an "ssh://" protocol as the "network server", and it says "Unable to find supported SSH command". So... it kind of supports sshfs? Except not? It knows what it is.

Input & Launcher

I could do with more oomph on the vibrate when you press the keyboard. It feels a bit vague. It's more noticeable when you're holding it in your hands than when it is sitting on the table.

The space bar is very close to the little arrow at the bottom of the screen that loads up the app launcher. So if you are not careful then it pops up the app menu while you're typing.

The app launcher only uses a third of the screen if there are any apps open, this is annoying. The top third of the screen is wasted on some sort of alt-tab-style view, with no way to get rid of it other than closing all the windows. The bottom third is wasted on a keyboard for typing into the search box, so you're left with only 2 rows of icons visible.

"Parental Controls" doesn't work. It shows a splash screen, spins for a couple of seconds, and then closes.

The Settings tool has an obvious option for whether I want my primary mouse click to be the left button or the right button, but no apparent option at all for how much haptic feedback I want from the keyboard.

Backups, again

Oh, there is a helpful help icon in the "network server" box on the backups thing that tells you what the URI should look like. It is a question mark inside a rectangle, so for me it pattern-matched "this unicode character does not exist in this font" rather than "click here for help". There is nothing to indicate that it is clickable despite the rest of the UI being the 3d style where it's obvious what is clickable and what is not.

The help text confirms that I typed my "ssh://" URI correctly, I wonder why it didn't work.

Aha, SSH isn't installed. In a terminal I get bash: ssh: command not found. That'll do it. I installed SSH.

Wow, it's working! It asked me for my SSH password, now it's having me setup some sort of password keyring. On the first try it tried to write to /pureos on the remote server, but after I changed it to /home/jes/pureos it actually worked. It backed up the phone to a network server without any rando Silicon Valley megacorp getting their grubby mits all over the data. Kudos, I did not expect that.

Telegram

There is no Telegram app in the PureOS store, however there are a few in apt. I wonder if they will actually work on the phone screen or if they're just normal desktop programs. I installed telegram-desktop with apt and it automatically shows up in the app launcher, which is a good sign, although it doesn't work.

Telegrar: Welcome to the officia. It's fast.

I mean, maybe it works, but all the input elements are off the screen and I have no way to scroll. I'll use the browser instead. Web Telegram worked on the first try, and there is even an option in the browser to install the site as an application, which adds it as an entry in the app launcher.

When I open it from the launcher it has forgotten my cookies so I have to reauthenticate, but after I've done this I can quit it and reopen from the launcher and it remembers me now. So this is satisfactory.

Terminal

There's no obvious way to type a tab character in the terminal. However there is a way to type a square root symbol, a pi character, and a tau character. Very useful in my terminal sessions.

Aha, found it. If you click >_ next to the space bar it takes you to a page of characters like Tab,Ctrl,Esc, etc.

Email

Email was completely uneventful. I loaded up Geary, typed in my account details, and it worked straight away. Great success, no complaints. And "remote images not shown" is default behaviour, as it should be.

Updates

Yay, now it has notified me that I have OS updates to install, one of my favourite tasks.

It's getting a bit warm. However, so is my other phone and I haven't even been using that one, very suspicious. I only used it to take the photo of the Telegram app. Maybe mere proximity to a Librem is enough to trigger remote surveillance in other phones?

It can't seem to work out whether it has downloaded the updates or not. There is a button that said "Download" at first, and then I click it, and it changes to something like "Update & Restart", but before I have a chance to click it it turns back into "Download".

Sod it, I'll use the terminal. I sudo apt upgrade and then reboot the phone. It takes about 9 seconds from unlocking the disk to fully booting the phone, very impressive. Admittedly it takes about 10 seconds to unlock the disk, not sure what that is all about.

Camera

The camera is very dark. At first I thought it had some debugging prints overlaid on top of the picture preview, but it turns out this is the camera UI. You click on the number next to "gain", "exposure", "balance", or "focus" (focus only available on the back camera) and you get a slider to adjust the corresponding camera property.

I took that with the Librem camera. Better quality than I had been led to believe.

The camera app was slow, it took a very long time to save the picture.

Hmm, I don't think it can record video.

I briefly entertained the idea that manually controlling the focus is good, because I frequently struggle to get my OnePlus One to focus properly, but no, it's not good, it's just annoying. It's more annoying than struggling with the autofocus on the OnePlus One.

Later in the evening I was showing off the camera to Emma, but by this time it was a bit darker, and this was the best I managed to do:

It's possible that I blundered the settings, but the camera definitely seems to struggle in low light conditions.

Maps

The pinch zoom on the maps app is really laggy. It also doesn't seem to understand postcodes.

It can do routing on the map, but the UI disappears off the right hand edge of the screen so there's no way to confirm that you want to start following the route. If you unlock screen rotation and make it landscape it no longer disappears off the right edge, but now it disappears off the bottom edge. Has anybody tried to use this?

Mobile network

I put a SIM card in the phone and opened up the phone app. It says "Can't place calls: No modem or VoIP account available". At first I thought this was just the natural order of things, because I thought calls didn't work, but Emma suggested rebooting it to see if that fixed it. I played along, but I knew it wouldn't help.

OK, I rebooted it and now it is showing a 4G symbol in the top bar. Could it be...? Wow, calls work! In both directions. Worked on the first try, incredible technology. Just make sure you reboot so that it knows you've installed a SIM card.

It makes a sound effect when an SMS is received, this is bad. I can't find an option to turn off notification sounds short of turning the volume down to 0. It sounds like if I ever listen to anything, I need to remember to turn the volume down to 0 afterwards otherwise my phone might be able to make noises. I don't like it when things make noises.

There doesn't seem to be a way to bulk-import contacts. Maybe this is a terminal window jobby.

4G worked after I typed in the APN settings for giffgaff. Hosting a WiFi hotspot works. It seems to take a long time to successfully switch from 4G to WiFi, I don't know why. Once it is connected it works fine, just seems to take a couple of minutes to connect to WiFi once you've switched 4G off.

Browser

The browser has no "stop" button.

YouTube is popping up its first-run "you need to agree to our terms" modal, but all the text and buttons are invisible. I tapped around a bit and it's worked now. I'm in YouTube. Video playback on YouTube works first try, great success.

Ah, but fullscreen doesn't do anything. It just stays non-fullscreen.

I wonder if Slack works. I'm guessing not because Slack won't let you use it if you don't have an authorised User-Agent string. Indeed, it doesn't work.

I can't open links in a new tab. Long press pops up a menu that includes the option, but in order to select the option I need to let go of the screen, and when I let go the menu disappears. Holding my finger on the screen and dragging it to the option also doesn't work.

Firefox

Graham told me I can get Firefox with the firefox-esr package. I did, and Slack works in Firefox, albeit very slowly.

Opening links in a new tab works in Firefox. Fullscreen video works in Firefox.

Ben remarked that it is weird that the default browser isn't just Firefox.

Conclusion

I'm glad the phone has finally arrived. I think it's a lot better than most people are saying. From reading the r/Purism subreddit you'd get the impression that the phone doesn't work and everyone is asking for a refund. It's certainly better than my OnePlus One, with the exception of the camera, and possibly the battery life.

I very rarely take my OnePlus One anywhere with me unless I know I'm likely to need it, because I reject on principle the notion of carrying an always-on tracking beacon that logs my movements in the phone companies' databases. With a hardware kill switch for the baseband modem, I can now carry the Librem 5 around without having to care about that, and I'm able to turn it on as and when I want to make a call or use 4G.

Maybe I'll try to fix some of the little annoyances with the phone (the "please take me to the launcher" arrow being mere pixels away from the "please input a space" button seems likely to be an easy fix). And hopefully the battery life isn't too awful.

If you want to 3d print it, you can download it from prusaprinters.

I don't know if you're familiar with Stewart Coffin's puzzle "Four Fit" (aka "Martin's Menace" because Martin Gardner said "It looks easy but is fiendishly difficult. I wasted a week trying vainly to solve it"). There is a 3d-printable version of Four Fit on thingiverse.

Four Fit is so difficult because the solution involves placing the pieces so that they are not aligned with the apparent axes of the box you need to put them in.

There is also a "Five Fit" puzzle, available for example here.

However I have not yet been able to find an example of a Six Fit puzzle, so I wrote some code to find one for myself, and I found this one.

Searching for puzzles

My "packominoes" tool generates grids skewed away from the axes. It searches for possible skewings where a section of the skewed grid intersected with an axis-aligned bounding box leaves a given number of squares available. Having found a suitable skewed grid, another program brute forces the set of all sets of pentominoes to try to find a set of pieces which can pack into the skewed grid, but which cannot pack into the largest axis-aligned grid that fits within the axis-aligned bounding box.

A generated puzzle, then, is an axis-aligned bounding box and a set of pieces.

Here's a diagram showing how the skewed grid and the bounding box interact for Six Fit:

There's a bit more info in my packominoes github repository.

More puzzles

I expect it would also be possible to find a Seven Fit and possibly Eight Fit, although the chances get slimmer as the numbers get higher because the number of possible ways to fit the pieces together start making it impossible to find a box that permits a "diagonal" solution without also permitting an "orthogonal" solution. Hope that makes sense.

We could also search for new variations for low-numbered packing puzzles of this type, using different sets of pieces.

And here's a video of surfacing the face of the gear so that the chuck can be mounted square:

I made this rotary axis because I have a new puzzle design which I want to make out of metal, and it requires machining all the way around a cylinder.

Why not a traditional 4th axis?

A 4th axis is normally mounted on the table, and controlled by its own stepper motor, indepdendently of the other 3 axes:

I initially wanted to build a real 4th axis, but decided it was too much trouble, because Grbl only supports 3 stepper motors. There are Grbl derivatives which can do more, but they require more capable hardware than an Arduino Uno, and I can't be bothered rejigging all of the electronics. So my next thought was to use the Y-axis stepper driver to control a rotary axis. When I wanted to use the rotary axis I would unplug the Y motor and plug its cable into my 4th motor. The downside is that I would still need to acquire a 4th motor.

Then one morning, while lying in bed, I was pondering some more on how to do this. I had some ideas about disconnecting the gantry from the leadscrew, and adding a bunch of bevel gears, with a shaft driven off the leadscrew, running vertically up through the table, but then it occurred to me that I could leave the gantry connected to the leadscrew and control the rotary axis with the gantry's movement relative to the table! This seemed like much too weird an idea to pass up, so I was compelled to implement it.

I wrote a few paragraphs about the idea to some friends, and in a post on reddit, but mostly people either didn't understand or didn't think it was a good idea. One person linked me to a Perry Projects blog post from 2017 with the same idea, which was quite encouraging.

Design

I took some measurements from the machine, modelled the important pieces in CAD, and then designed the rack, gear, and mounting plates to match:

The 3 bolt holes in the mounting plates are deliberately oversized, to provide wiggle room for alignment. I need to be able to slide the plates up and down to ensure a tight engagement of the gear teeth (I don't want any free play at all), and I need to be able to slide them forwards and backwards to ensure that the spindle is centered on the centre of the cutting tool.

The gear has a step in it to fit the recess in the back of the 80mm 3-jaw chuck from my mini lathe.

The shaft is made out of what's left of a pneumatic cylinder I made a few years ago, it was chosen because I have already cut a thread on one end, and it was almost the right diameter to suit 2 large bearings that I already had.

Finally (not pictured) there is a 3d-printed spacer behind the gear to provide clearance for the nuts holding the chuck, and a 3d-printed "bung" to screw up the end of the shaft to prevent it from falling out. Ideally there would be a spacer between the 2 bearings but I had already pressed them in by the time I thought of this. If this proves problematic I can always knock one of them back out and fit a spacer in future.

Build

Making the plywood parts is easy, it's just normal CNC work. They're made out of offcuts of Russian Birch plywood that I bought for my SCAMP case. I really like this material, it leaves a nice crisp cut with very little tearing. There are only 5 parts: the gear, the rack, and 3 identical plywood plates, with a bearing pressed into each of the outer plates.

The gear needed 2 setups (top side and bottom side), because there is a step in the front to fit the chuck, and there are also recesses in the back for the nuts to sit in. In hindsight I could have done it in a single setup: I could skip the step in the front because I needed to re-cut this later anyway.

I don't have any pictures from making the shaft. It is a steel tube with a female thread on one end, turned down on the lathe to 25mm and lightly polished. It's worth taking your time getting this right as you don't want any play between the shaft and the bearings. I have a very agricultural "moving steady-rest" that fits this tube (which you can see in action here), but I still got quite a lot of chatter, so polishing the surface was really necessary.

Most of the squareness requirements are provided "for free" because the plywood surface is already flat (enough), and the gantry is already parallel (enough) to the Y and Z axes of the machine. The only tricky part is making sure the shaft is properly perpendicular to the gear. If it's not, then the gear will wobble as the shaft rotates and so will the work piece.

I didn't have a good plan for fixing the gear to the shaft. I just dripped a load of superglue in the hole and then pressed the shaft in with the vice. It is a good tight fit and seems secure, but unfortunately it was about 1° out of square, which was very disappointing, and I didn't immediately know what to do about it.

As I left the 3 plywood plates gluing up overnight, I realised that I could use the CNC machine to square up the face of the gear for me! I just need to drive the gantry back and forth with the cutting tool up against the gear, and it will automatically cut it square. What a relief! In some sense it was a blessing that I failed to glue the gear square to the shaft, because it forced me to come up with this alternative and superior method.

Next we need to bolt the mounting plates to the gantry. I cut out a cardboard template and aligned it with the approximate required position of the mounting plates, marked the holes on the gantry, and drilled them:

I drilled the gantry holes 8.5mm to accept M8 bolts, with the idea that I could drill this out to 10mm or 12mm if I need more wiggle room for alignment, but in the end I didn't need it, the oversized holes in the mounting plates were enough.

The rack is just screwed straight down to the table:

I was able to fit everything together so that the gear is nice and tight in the rack, and there is no discernible play in any direction, which I'm very happy with.

An unexpected benefit of making the shaft out of a hollow tube is that you can sight down the bore to check how it is centered on the cutting tool:

With the shaft installed, I surfaced the gear (shown in the video at the top) and fitted the chuck:

Next

Obviously the next step is to use this to make an actual part. I've only just finished building it, and so far the only cuts I have made were to surface the faceplate. I now need to write some software to turn my puzzle parts into G-code, and then I can try to make one out of a piece of plastic pipe, and only when that is successful will I try to make one out of aluminium or brass.

I'm not sure if I'm going to need tailstock support. Perry's Project uses a wheel that rolls along the table to support the far end of the work piece. Maybe I would do something similar:

Limitations

On a "normal" 4th axis, you can spin the part around and around forever, but with the one I have built there is a very limited range of motion, because if you try to move the Y axis too far you fall off the end of the toothed rack. This also means that we can't easily use traditional 4-axis CAM software, because it doesn't know that it has to "rewind" periodically to stop itself from running out of bounds. But my CAM software of choice is FreeCAD, and that doesn't support a 4th axis anyway.

We also can't do things like using this as an indexing head to machine flat features on different sides of a part, because the cutting tool is always centered on the axis of rotation.

Useful resources

If you want to learn about building rotary axes, you might enjoy watching This Old Tony's 4th Axis Build and CNC Lathe videos.

If you want to cut a toothed rack, then KHK Gears have a very useful page on the involute gear profile, including a diagram with all the important dimensions explained (apart from s and e, which I gather you adjust to get the tooth clearance you desire?):

It's called "Party Puzzling" and we have a website at partypuzzling.co.uk and an Etsy shop at https://www.etsy.com/uk/shop/PartyPuzzling. Right now we're charging £7.50 per puzzle. They cost about £2 in materials and I can only produce 8 per day, so we're not going to get rich off this any time soon. But I still think it's a fun idea.

We first did this for our own wedding, with chess-themed puzzles, and people liked it, which is why we're making them available for sale. We're offering Easy, Medium, and Hard difficulty options, although I still haven't completely finalised the difficulty calibration. Every puzzle in the same order will get the same maze.

The puzzles consist of a maze on the inside of a cylinder, and a peg on the inner piece which follows the track of the maze. To get the inner piece out you need to solve the maze, which is tricky because you can't see it.

So far we only have the beer bottle theme up for sale, but the chess piece theme is mostly finished and will probably go up on Etsy soon. If you have any good ideas for puzzle themes please let me know! A good theme would be a well-known object with a roughly cylindrical shape, with a sensible "parting line" to allow the 2 halves to rotate.

Maze design

The thing that makes hidden mazes hard for most people is that they don't know the general technique for solving them. Most mazes can be trivially solved by simply holding your left (or right) hand against the left (or right) wall, and following the left (or right) wall until you reach the exit. This works even if you have no way to find out whether you've already visited a given point in the maze. You don't need to remember anything, it just automatically follows the walls all the way to the exit.

If you didn't already know this algorithm, perhaps you want to convince yourself of its correctness. Here's an easy example you can try it out on:

You should be able to see that by following one of the walls you expect to explore 50% of the maze before you find the exit. If you can't see any more than your immediate surroundings (which you can't if the maze is invisible) then no algorithm can do better than this in the general case (I claim but do not prove).

You might think (as I did, before I thought about it enough) that this only works because there are no cycles in the maze. You might think that if we add a cycle, then it is possible to get trapped on the cycle:

Not so. Far from trapping the player on the cycle, you actually isolate the player from the cycle! There is no way to add a cycle whose walls are connected to the outside walls of the maze (otherwise it's not a cycle), which means there is no way for a player who is following the walls of the maze to reach the cycle in the first place. (Although it is true that a player who reaches the cycle by some other method, and then begins to follow the left-hand-wall algorithm, will indeed remain trapped.)

However, putting the maze on the inside of a cylinder gives us a distinct advantage! Represented on a 2D drawing, the left-hand edge of the maze wraps around to the right-hand edge. If we make a cycle that wraps all the way around the circumference of the cylinder, then we can trap the player in the cycle, and isolate them from the rest of the maze:

So we can use this trick to make the mazes challenging even for people who know the left-hand-wall algorithm :).

Build pipeline

Manually designing cylindrical mazes in FreeCAD is really annoying, so I have quite a fun build pipeline.

I first draw an ASCII art maze in vim, a simple example would look something like:

- ----------
|         | 
- --- - --- 
    | | |   
------- --- 
| |     |   
- ----- --- 
        |   
- --- - ----
    | |     
- ----------
            
------------

I then have a Perl script which reads in the ASCII art maze and produces an OpenSCAD source file that produces a 3D model of a flat maze:

To save time, I also have a Perl script which can generate the ASCII art mazes, and another script that can take in a maze and give a number for how complicated it is. I run these 2 together in a loop to search for mazes of the appropriate difficulty. Finding a metric for how difficult it is to solve a maze is quite hard, so there's still a lot of manual "curation" here. A key part of the measure is the question:

• How many places does the player have to make a turn that is not at the end of a corridor? I.e. how many times do they have to preemptively attempt to turn a corner without any tactile feedback that a corner exists?

Having got an ASCII art maze ready, I have a rust program that reads in an STL file and wraps it up into a cylinder. This works by splitting large triangles up into tiny ones, and then rewriting the coordinates by interpreting them as Polar coordinates and rewriting them as the Cartesian equivalents:

• The x input coordinate gives an angle of rotation around the z axis
• The y input coordinate gives a radius away from the z axis
• The z coordinate is unchanged

And I can run a command like mkpuzzle beer easy-maze which will use the above steps to generate an STL file from the ASCII art in easy-maze, and merge it with the manually-modelled outer part of the beer theme, ready for 3D printing.

Online leaderboards

For our wedding I generated a load of random hex strings to use for the leaderboard codes. I thought it would be funny if there was an extra layer to this, so instead of just choosing random hex strings and putting a list of them in a database, I made the rule that all strings with a SHA256 sum beginning beef42... were accepted as valid codes. With the idea that if anyone went to the trouble of finding out the SHA256 sum of more than 1 code, then they might get some joy out of making a "keygen" and submitting hundreds of falsified entries to the leaderboard.

Unfortunately, restricting the search so tightly led to me creating a handful of duplicate codes without realising it, so a couple of the guests found that when they solved the puzzle and tried to scan the code, it was rejected. Egg and my face were in alignment!

So for Party Puzzling I am just generating a bunch of random strings and putting them in the database.

The motivation is that Martin wanted to use the Psion to display notifications, but couldn't find anywhere to buy the "CommsLink" module that connects it to a PC. I thought it sounded like a fun Arduino project, so I did it.

This is what the Psion looks like, with my homemade comms module plugged in:

My code and 3d printing files are all available on github.

If you are working with one of these devices then you should see Jaap's Psion II Page. It has lots of good information and documentation.

Hardware

The Psion actually has 3 slots that peripherals can be plugged into. The first 2 are on the bottom of the device and are used to insert "datapaks". These are UV-erasable EPROMs which can be used to store code and data. The Psion has a weirdo "write-only" filesystem built on top of these EPROMs, which (I assume) "deletes" files by writing a bit to say that the file is no longer there, but can not reclaim used space. The only way to reclaim the space is to expose the little die window to UV light and erase the entire thing:

The third slot, the "top slot", is the one we're going to be using. It has 16 pins. The important pins are D0..D7 which are connected directly to the CPU's "port 2" data bus. There's also ground, +5v, and various other signals which we can ignore. Jaap has a page on top slot technical data

My first thought for the easiest way to write bytes from the Arduino into a program running on the Psion is to just assert the byte onto the port 2 bus and leave it there permanently. Then the program on the Psion can sample the bus at its own convenience. When we have another byte to send, we assert that byte instead, and the Psion will know it's a new byte because the value has changed. If we need to be able to write duplicate bytes then we can stick a 0x00 in between.

Unfortunately, this isn't quite enough, because it prevents the datapaks from working. All 3 slots are connected to the same bus of the CPU, but with 1 pin that tells you whether that slot is selected (CS1, CS2, CS3). Most of the pins are identical between all 3 slots. If you try to assert something onto the bus while some other peripheral is trying to use it, then you get bus contention, which means in the best case the computer doesn't work properly, and in the worst case is permanently damaged.

So what do we do about this? The ideal solution is to put some glue logic between the Arduino and the organiser that tests the CS3 pin (and "output enable" preferably) and only tries to assert anything to the bus when the top slot is selected, and the CPU is trying to read from the bus rather than write to it. (It's not sound to do this logic in software on the Arduino because even at 16x the clock speed of the Psion, the Arduino is not fast enough).

But since we know that the only other devices connected to the bus are the datapaks and the CPU, and we know that those things all respect the control signals properly, and we only need unidirectional communication, we can take a bit of a shortcut. I connected some digital outputs of the Arduino to the data pins via 510 ohm resistors (chosen mostly at random). This is low enough resistance to overcome the internal pullups on the data lines, but high enough that it doesn't interfere when the datapaks or CPU are trying to write to the bus. It's a really simple solution and seems to work well. So that's the hardware problem solved, the rest is just software.

I initially got this to work on a breadboard, but built it up on a stripboard once I knew it worked:

Connecting a 2-row pin header to a stripboard is not easy because there is no sensible way to connect to both rows. I solved this by soldering the connector on one side and supergluing it on the other:

(And, actually, I didn't even have a 2-row header on hand, so I made this one by gluing two single-row headers together).

Software

We need to be able to read from port 2 and write to the display. The Psion has a programming language called "OPL". It's more or less the same as BASIC. Writing to the display is easy, because we can just use PRINT and have OPL sort it out for us. Reading from port 2 is more tricky.

Jaap's page on The System Variables is very helpful. We learn that port 2 is accessed at memory address 0x0003, and the direction (read or write) of the pins is controlled at address 0x0001:

Port 2 data direction register. Bit 0 controls the direction of bit 0 of [port 2] (1=output,0=input) and bit 1 control the direction of bits 1-7 of [port 2].

So we need to write zeroes into address 0x0001, and then read bits from the data pins in the top slot by reading address 0x0003.

It turns out we also need to tell the organiser that we want to use the top slot rather than one of the other slots. I'm not sure if one of the datapaks is left active if you don't do this, but I couldn't get it to work without asking for the top slot to be active.

We control which slots are active by writing to port 6. See Jaap's page for more info.

OPL has a POKEB command which can be used to write bytes into arbitrary addresses, and it has a PEEKB() function which can be used to read from arbitrary addresses. Great success, right? We just poke and peek and we can read from the top slot.

Not so fast! Just to make life more interesting, OPL deliberately blocks any attempt to read or write an address below 0x0040. This is very frustrating. It's probably done to "prevent you from breaking your device", but a.) if you're writing values into random memory addresses you should already have some idea of the risk, and b.) you can still break your device anyway with the workaround.

The workaround is to write some assembly language code to do your memory accesses, assemble it by hand, write the bytes into a string, and then execute the string from OPL as if calling a native machine code routine (which you are). Easy enough, but it's annoying that it's necessary. It wouldn't have been any trouble for them just not to block access from OPL! (In fact, come to think of it, maybe the easiest way to circumvent it is actually to work out where in memory POKEB and PEEKB() are implemented, and NOP out the test...)

I was very excited when I managed to use some machine code to read data from the top slot, then get those bytes back to OPL, print them to the screen, and observe that they changed when I pulled one of the pins on the bus high:

And then even better when I first got the complete pipeline working, getting text typed on Linux displayed on the Psion:

I eventually reached the following machine code routine to set port 2 to what I want, read a byte from it, stick the byte in the X register, and return (in my made-up assembly language dialect):

72 18 16  oim $18, $16 # allow writing to bits 0x10,0x08 in port 6
71 18 17  aim $18, $17 # enable the top slot and disable the other 2 slots
71 00 01  aim 0, $01 # input mode on port 2
96 03     ld a, ($03) # read from port 2 into A
c6 00     ld b, 0
36        push a
37        push b
38        pull x # make a 16-bit value out of a 0 and the byte we read
39        rts

I wrote this by consulting Jaap's HD6303 instruction set documentation.

I have never programmed a HD6303 before, so some of this is probably stupid (like spending 4 instructions to extend a byte in A to 2 bytes in X), but it does the job. With machine code in hand, we just have to write an OPL program to execute it.

I ended up with this:

DISPLAY:
LOCAL S$(17),A%,B%,C%
POKEB $7C,0
S$=CHR$($72)+CHR$($18)+CHR$($16)+CHR$($71)+CHR$($18)+CHR$($17)+CHR$($71)
S$=S$+CHR$(0)+CHR$(1)+CHR$($96)+CHR$(3)+CHR$($C6)+CHR$(0)+CHR$($36)
S$=S$+CHR$($37)+CHR$($38)+CHR$($39)
WHILE KEY<>0:ENDWH
WHILE KEY=0
A%=USR(ADDR(S$)+1,0)
IF A%<>B%
C%=USR(ADDR(S$)+1,0)
IF A%=C%
IF A%<>0
PRINT CHR$(A%);
ENDIF
B%=A%
ENDIF
ENDIF
ENDWH

The local variable S$ holds our machine code. Strings in OPL have 1 byte saying how long they are, followed by the contents of the string. ADDR(x) gives us the address of a variable, and USR(addr, arg) executes some machine code. So we can execute our code by calling USR(ADDR(S$)+1,0). I didn't figure this out on my own, I got it from Jaap's machine code tutorial.

The initial POKEB $7C,0 asks the Psion not to switch off the screen after a period of inactivity. The main loop is inside WHILE KEY=0 so that you can exit it by pressing any key.

The main loop reads a byte from the top slot. If the value is different to the last known value, then it reads again. If this second read is different, then it ignores them both. This is so that we don't get corrupted reads while the Arduino program is halfway through updating the values on the bus.

Having determined that 2 consecutive reads got the same value, we print the byte to the screen unless it's 0 (which is used to delimit consecutive outputs), then we update the last known byte to whatever we just read and loop again.

Although the current implementation is only unidirectional, it is probably possible to invent a protocol by which the "writer" can signal to the "reader" that it wants to swap roles, and then both sides invert their bus direction. I'm not sure if we'll need to do this, but hopefully it would only require a software change.

3D Printing

The last thing to do is make a nice plastic enclosure for the electronics. I recently fitted a 0.6 mm nozzle to my Prusa Mini (previously I was using the standard 0.4 mm nozzle). I think this is a great upgrade. You can print parts a lot faster, they're (supposedly) a bit stronger, and the nozzle doesn't clog as easily because larger impurities can pass straight through. The loss of detail is minimal, and only exists in the X/Y plane because you can still print with the same layer heights if you want. I think 0.6 mm should be the default!

I really like countersunk allen screws, I think they give a nice finish sitting flush with the surface of the part.

If you want to try it out, you can get it on my fork of the BangleApps loader, or, eventually, on the official BangleApps loader. Read the README if you want to know more.

Here's what it looks like mounted on my handlebars with a 3d-printed "artificial wrist":

It was inspired by the Beeline device which works exactly the same way, except is more expensive than a Bangle.js and you need to use their proprietary smartphone app to set up the waypoints.

Test ride

On Tuesday I tried to use Waypointer Moto to navigate to Cheddar Gorge. To cut a long story short I didn't get to Cheddar Gorge, because I came across a road closure about 2 miles into a windy country road and had to turn back. But I did discover Ebbor Gorge just off said windy road, which I never would have come across if I were following a normal route and is quite a nice walk, so in that respect it was a great success.

But if you're really keen to get to where you want to go, maybe this isn't the best way to do it.

Development

Waypointer Moto is based on the "Way Pointer" app for Bangle.js, but with a few important changes:

• Use the GPS course instead of the compass to work out the direction of travel, because the compass is unreliable. Way Pointer uses the compass because Way Pointer is designed for walkers, and the GPS course is rarely available if you're travelling at walking pace.
• Show the status of the GPS fix in the colour of the arrow (red for no fix, yellow for location but no course, white for all good).
• Add an OpenStreetMap integration to the PC-based waypoint editor so that you can click the destination on the map instead of having to manually type in latitude/longitude coordinates.

(and a number of less important changes).

As always, developing for Bangle.js was really easy. The only annoying part is adding the integration with the BangleApps loader, because you need to wait for GitHub actions to rebuild your GitHub Pages every time you want to see if your change has worked.

Plans

To make Waypointer Moto more useful for situations where you actually really do want to get to where you're going, I plan to add a "route" mode where you click several waypoints along the route to your destination and it will automatically step from one waypoint to the next as you approach them.

I also want to add a charge connector to the "artificial wrist" so that the watch can get charged up while it's in use. I think leaving the screen and GPS on 100% of the time will drain the battery quite quickly. On Tuesday it drained about 50% in 2h30, so I expect it would only last about 5 hours without charging.

A motorcycle that's leaning but not cornering looks like this:

The weight of the bike and rider unit is not positioned above the contact patch of the tyres, so the centre of mass is going to accelerate downwards and the bike will crash. (Alternatively, the weight and the reaction force from the ground are not aligned, leaving a resultant torque, so the bike is going to rotate and crash.)

When cornering, we have the centrifugal force counter-acting the lean angle (or, if you prefer, we lean in order to counteract the centrifugal force):

(Reaction arrows omitted for brevity).

A 1-axis accelerometer measuring vertically in the bike's frame of reference would see acceleration corresponding to the red arrow.

Plotting this as a triangle (and turning the forces into accelerations) we get:

(v²/r unknown, g assumed constant).

Which we can solve with trigonometry to find the angle:

cos(θ) = g/a
θ = acos(g/a)

In reality this would not work quite so well (even if our accelerometer were perfect, there's no vibration, etc.) because:

1. We assume the forces are balanced. In steady state (constant speed, constant radius) this is true, but you need to temporarily unbalance the forces to make the bike lean. We would under-estimate lean when the bike is tipping into a corner and over-estimate when it is standing back up.
2. We assume a flat road. If the bike is under acceleration in the vertical axis (e.g. cresting a hill) then the effective value of g for the purposes of our triangle is incorrect. We would over-estimate lean in troughs and under-estimate on crests.

This is what it looks like:

It's relatively common to write Advent of Code solutions to run on limited hardware. What isn't so common is to develop on the limited hardware, which is what I've been doing, even to the point of reading the problem statements and submitting the solutions via a proxy running at the other end of a serial port. The main difference is that exploratory programming is much more difficult in an environment where you can't just write a quick Perl script to learn something about the shape of the problem. Even compiling a small program takes over a minute.

So far I've completed part 1 of every day except day 24, and part 2 of every day except 15, 22, 23, and 24, although on a couple of days I did resort to using the emulator (~20x faster) when the physical hardware got too frustrating.

Videos

I recorded video of a lot of my attempts. I think the day 6 video is a good one to start on, because it is relatively short and easy to understand, and I didn't make too many blunders. You might still prefer to watch on 1.5x speed.

• Watch my day 6 video »

I enjoyed doing the videos, and a handful of people told me they enjoyed watching them, so I think I'd like to do more programming videos in the future.

Notable solutions

Day 4: Giant Squid [link]

I had already been stung a couple of times by programs that took several minutes just to parse in the input, so I had the idea that I could leave the input somewhere high in memory that nothing else is going to overwrite, and then just grab the input data straight out of memory without having to parse it.

So for day 4 I did exactly that. I wrote a program called read.sl that read integers from stdin and wrote them into memory starting at address 0xc000, and wrote the number of integers into address 0xbfff. Then part1.sl looks in 0xbfff to see how many integers there are (with an assertion that this number is 2500, so that it can notice if they are overwritten), and then operates on the input in-place starting at 0xc000:

var input = 0xc000;
var inputlen = *(0xbfff);
assert(inputlen == 2500, "inputlen = %d\n", [inputlen]);

It did work but I think it may have been more trouble than it was worth. It turned out that parsing the input for this day wasn't all that time-consuming, and it adds extra stress of trying to make sure you don't do anything that will silently overwrite your input data. It's a fun idea though, I'm glad I tried it out.

Day 5: Hydrothermal Venture [link]

This was the first day where my 64K words of memory presented a problem. You're operating on a grid of 1000x1000 points. A million points is too many points to fit in 64K addresses. Even if you only use 1 bit per point of the grid, that's 62500 points, leaving only 1500 addresses remaining, which isn't even enough for the kernel, let alone the program to solve the problem.

Ben suggested splitting the input up into a number of smaller subgrids, and then solving the smaller grids consecutively. This was a pretty neat solution, and it worked fine. The only caveat was that splitting the input up into smaller inputs was more complicated than the actual problem! But that's all good fun. And at least it fits in RAM. That's exactly the kind of challenge I was looking for when I started doing Advent of Code on SCAMP.

Simon subsequently emailed me a solution to part 1 that runs in a single pass over the input file by sorting the input lines by x coordinate, and only looking at a single x coordinate at a time and testing which lines overlap in the y axis at that point in x, which is pretty cool. I believe he is the 2nd person ever to have written a SCAMP program.

Day 13: Transparent Origami [link]

I used a hash table as a sparse 2d array of points in this problem. Initially I wanted to use sprintf("%d,%d", [&x,&y]) to format the keys for the hash table, but I found this was running out of memory much more quickly than expected. I then realised that since my "characters" are 16 bits wide, and the integers that I'm formatting into the string are also 16 bits wide, I may as well just write the integer directly into the string! As long as I don't need to handle 0, this works fine and every integer fits in a single character. It's a much more efficient way to generate hash keys, because there's no time spent on formatting integers, and less memory wasted on storing characters. This is a good trick that I used a couple of other times as well.

I made sure that values of 0 don't prematurely terminate the string by adding 1 to every coordinate:

var k = malloc(3);
k[0] = x+1;
k[1] = y+1;
k[2] = 0;

After completing day 13 I had a look at my sprintf() code and realised that it initialises the output string to 64 words long, growing it as required. This is too much in applications that format large numbers of short strings, so I've reduced the starting length from 64 to 8 words. But just sticking integers directly into the string is still better.

Day 17: Trick Shot [link]

After reading the problem statement I still didn't have a great intuition for what the trajectories looked like, so instead of writing a program to find the answer, I first wrote a program that would let me input candidate velocities and see whether the probe ends up flying over the target, under the target, or hitting the target.

I discovered quite quickly that once the y velocities are high enough, there is only 1 x velocity that results in hitting the target, because of the "drag" effect: the x velocity needs to be reduced to 0 at a time where the x position is within the target range.

I manually explored the search space and found a convincing-sounding candidate velocity, submitted it, and it was the correct answer. So I didn't write a part 1 solution at all!

Day 19: Beacon Scanner [link]

This was quite a big problem for SCAMP and my solution took several hours to run even though it was quite well optimised and (I believe) asymptotically optimal. I used the trick from day 13 of generating hash keys by sticking integers directly into characters, but went one step further by writing the function to do this in assembly language as this function is arguably the "innermost loop" of my program.

Day 22: Reactor Robot [link]

On part 1 you're operating on a 101x101x101 cube: that's just over a million points, which is too big to fit in memory. I reused the day 5 idea of processing the input in smaller chunks. I used a bit-set to represent the cube, and processed it in 4 quarters consecutively, so I only need about 16K words of memory per quarter.

For part 2 even this isn't enough. The correct solution is to use coordinate compression to remove all x, y, and z values that are not present in the input data, then operate on the much smaller grid, and then multiply the number in each cell of the smaller grid by the number of "real" grid cells it represents.

I started working on this but ran out of time before I got it working.

Day 23: Amphipod [link]

I solved part 1 by hand in my text editor, without writing any code at all, which was fun.

I tried to do the same thing for part 2 but gave up and started writing a program to solve it, but I ran out of time before getting it to work. But I later learned that Owen solved part 2 by hand, so I have had a couple more attempts at this but still can't do it. I can't even find any way to get all the amphipods into their correct homes, let alone the shortest way! I read on the subreddit that there is a 40x difficulty range between the easiest and the hardest inputs, so maybe I just got unlucky. That's what I'll tell myself.

Problems still to solve

Day 15 part 2

This is a shortest-path search on a 500x500 grid. The grid is derived deterministically from the 100x100 grid that is used in part 1, so lots of people on the subreddit are talking about how they cut down their memory usage by not putting the full grid in memory, but I think this isn't enough. Yes, it is easy to derive the edge cost for a given edge without storing the full grid, but the queue for Dijkstra's algorithm is still going to be too big to fit in memory, and you still need 1 bit per vertex for the visited set so that you don't re-visit previous points.

My plan is to put my priority queue and my visited set on disk, and use most of RAM as an LRU cache of disk blocks. My filesystem only supports 65536 blocks, so everything after the 65536th block is invisible to the filesystem, but my CompactFlash card is much larger than this. In LBA mode, one 16-bit word sets the upper bits of the block number, and another sets the lower bits. I can copy and paste the block IO code from the kernel, but just set the upper bits to some nonzero value, and then I can treat the CompactFlash card as a random-access block device rather than having all the overhead of my filesystem.

So once the input file has been read into memory, I'm not using the kernel to do disk IO. The only thing I still need from the kernel is console output, which is also easy to implement directly in my program, and then I don't need the kernel at all! This gives me about 48 more blocks' worth of disk cache, at the cost of having to reset the CPU when the program is done, instead of using exit() to return to the shell.

I am concerned that this will be too slow, but it sounds really fun to work on. I hope to solidify the design on paper and then film a video of my implementation. I think I'd like to output a 2d progress visualisation to the console, showing which parts of the grid have already been explored by Dijkstra's algorithm.

Day 22 part 2

I expect that I just need to implement the compressed coordinates version of my part 1 solution (which I already started), and go to some effort to minimise the number of bigint operations.

Day 23 part 2

I think the way to go here is to come up with a sensible representation of the map and then solve it with Dijkstra's algorithm. I started doing this, but my representation of the map was too convoluted. There are actually only 7 places that the amphipods can walk to other than their final destinations, so treating it as a generic grid that they can freely walk around over-complicates it quite badly.

Probably I want each of my nodes in the search space to represent the 7 locations the amphipods can park in, and a stack for each of the "rooms" that they end up in, and I'll want a function to calculate the path cost between a pair of points.

Day 24

I tried to solve this by reverse-engineering the input and working out what to do to make z == 0, but only managed to convince myself that it is impossible unless I can input a single digit with value 15.

I will probably have to read the subreddit carefully and try to pick the good ideas that other people have posted. Graham views the z register as a "stack" encoded in base 26, which is either pushed to or popped from depending on the value of each digit.

This makes sense, and I think the problem with my attempt at solving it by hand is that I greedily popped at every stage I could, whereas you actually sometimes need to push even if you can pop, just to ensure that you'll be able to do enough pops later on.

It ended up easier to solve than I intended, because the bubble parts were printed in the "solved" configuration, which means the 3d printing lines all point in the same direction, so it's easy to know which way they should be oriented.

If you want to print it yourself, you can download the print files from prusaprinters.org.

I haven't used prusaprinters.org before, but so far it seems vastly superior to Thingiverse: you can upload designs on the same day you sign up instead of having to wait 24 hours to be "approved", and the pages load like they're not served over dial-up.

In comparison to normal jigsaw puzzle geometry, in principle, this puzzle should be harder, because there's no clue as to the piece orientation (jigsaw pieces can only fit at 4 angles, circles can fit at any angle), and there's no clue as to which pieces form the outer border.

Design

I started by drawing a "master sketch" in FreeCAD, with a bunch of overlapping circles.

This sketch is used to model both the pieces and the casing that they fit inside.

For the pieces, I made a flat plate and then raised up the pieces with small gaps in between, in a single part:

Ideally each piece would be a separate part, but I couldn't work out an efficient way to do that in FreeCAD.

I think a taxonomy of the pieces would identify 4 distinct properties, based on the overlapping circle shapes:

1. pieces that are a complete circle
2. pieces with all sides concave
3. pieces with a sharp concave corner
4. pieces that form the outer border of the shape

I made sure to include at least one of each type.

I exported the single part with all the pieces as a single STL file and then used the "Z Cut" feature in PrusaSlicer to chop the bottom off, so that they would all be loose once printed:

To make the case, I just made a solid block and pocketed in the outline of the master sketch, with a gap around the outside:

I left 0.5 mm clearance between the pieces, but this turned out to be too much and they are too loose in the final puzzle.

Improvements

For a second iteration, the following changes should be made:

• the pieces are too loose: the clearance should be reduced
• the pieces should be separated from each other and printed in a random orientation
• the pieces are too small and fiddly: the entire puzzle should be scaled up by a factor of 2 or 3

It might be nice to make a larger version in wood using the CNC machine.

Another idea would be a 3-dimensional version, where instead of a bunch of overlapping circles with bits cut out, it would be a bunch of overlapping spheres with bits cut out.

Printed in Prusament PC Blend, it looks like this:

I would suggest printing this sort of thing in something like PC, TPU, or Nylon. PETG would probably be fine too. I know PLA not to be strong enough.

DJI's FPV drones have guards like this:

And this is a fine solution for a mass-produced part!

The weakest part of a 3d print is the joins between the layers. The problem with trying to 3d print the DJI shape is that whatever orientation you print it in, the cross-sectional area for layer adhesion would be very small at some point or another, which means they would break very easily. You can mitigate this by making the part thicker, but that would add weight, which is at a premium on a small quadcopter.

My design prints flat and is then folded up later, using only the existing motor-mounting screws for attachment.

(Image from Wikipedia)

It makes use of spherical geometry to form a triangle out of 3 angles that sum to more than 180°. Trying to close such a triangle causes the sides to bulge out to fit a sphere, which gives our shape the 3-dimensional nature we're after. I haven't actually got a solid grasp of the maths required to calculate the lengths of the sides so I just made some educated guesses and then refined the shape over a handful of iterations. (An arc through α radians with radius r must have length αr, but it is more complicated when dealing with real-world parts of non-zero thickness).

At 0.8mm tall, and printed in PC Blend with the "0.15mm QUALITY" preset in PrusaSlicer, these work out to only 0.9g each.

Here's a photo of an earlier iteration, printed in PLA, and showing a comparison between the printed part (flat) and installed part (curved):

By printing it flat, almost all of the stress on the part is contained within the layers, and the little stress that remains between layers is spread over the surface area of the entire part. This makes it significantly stronger than a comparable part that is printed in its final form. Another benefit of the flat design is that it is suitable for laser cutting or CNC routing.

As it stands it's definitely not robust enough when printed in PLA:

But I've had quite a lot of crashes with the version printed in PC Blend and it is not showing any signs of damage.

Using geometry to force flat parts into lightweight curved shapes seems like quite a powerful idea. I wonder what parts other than quadcopter rotor guards we could make lighter or stronger this way?

Design

A few months ago, Ruari had the idea of making a hollow die with a magnet under each face and a ball bearing inside. Using a larger magnet external to the die, you would attract the ball bearing to whichever face you wanted to be heaviest, and therefore make an adjustable loaded die. I did some experiments with that idea but I couldn't get an external magnet to be strong enough to move the ball around, and I couldn't stop the ball from moving around on its own when shaken.

I got this tiny linear servo off eBay:

At 20 mm long, it is only slightly larger than a typical d6.

I modelled a die, split into 2 parts, which I could mount the servo inside:

Three of the holes on the face with number 6 go all the way through. This will be useful later.

I also designed a small platform to mount on the servo horn, to carry the weight:

Build

With the parts printed out and assembled, and 3 short brass rods turned on the lathe, we get this:

So perhaps now you see how it works. The 3 wires that control the servo are soldered to the back of 3 of the brass pips.

At this point I realised that with the weight mounted on top of the platform, I would be able to drive it all the way to the face with number 1, but not really get it very close to the face with number 6. I want to be able to roll 6's on command, so obviously this needs to be the other way around. Right?

So I put the platform on the other way up before gluing the steel rectangle to it. Now the weight can get very close to the 6 face. Great success?

Except that's the opposite of what I want! If I want to roll 6's, then I need the weight to be opposite the 6, so that the 6 faces up when the weight is at the bottom. So it was right in the first place. D'oh! But with the platform upside down, the steel ends up not centered on the platform, so I can't put the platform back where it was else the steel won't fit inside the cube. Oh well. I'll just have to roll 1's on command instead of 6's.

At this point I lamented that the brass rods were taking too long to make on the lathe, and Dimitris suggested cutting them out of brass sheet on the CNC machine. I don't know why this didn't occur to me before, but it's much quicker. I cut out the remaining circles and glued them into the remaining spots for pips:

I realised another blunder at this point, which is that I managed to stick the pips for the 4 off-centre:

This is because I centered the pips on the origin in CAD, but the face was not centered on the origin, so the pips are not centered on the face:

Never mind.

Now it's time to tidy up the faces to make them look a bit neater, starting with the pips that connect to the servo, because they stick out the most:

And finally, after some sanding and clearcoat, we have this:

Which I'm quite happy with.

Cradle

Now we need a cradle to hold the die in to connect to the conductive pips.

The Arduino is running the "Knob" example sketch (File -> Examples -> Servo -> Knob), which reads the value from the potentiometer and sets the corresponding position on the servo. The wires that need to connect to the servo go to "pogo pins" that stick through the square recess in the cradle. These are sharp spring-loaded contacts that press against the pips.

I was initially planning to try to sand back the clearcoat on the pips to expose the conductive contacts, but this turned out not to be necessary because the pogo pins are sharp enough to reach the brass through the lacquer, which leaves small scratches:

Evaluation

Uniformity

I rolled the die 200 times in each configuration to see how much bias the weight gives the die.

With the weight in the middle:

It's not exactly uniform here, but it's really not that bad.

There is a slight bias towards the 3 because the 3 is directly opposite 4, which the servo is mounted on, and the weight of the servo means the 4 ends up at the bottom slightly more often than it should. An easy-ish solution to this would be to put a counterweight on the back of the 3.

And there is a slight bias towards the 1, I think because the range on the servo is not quite enough to drive the weight far enough away from the 6. Probably this would be improved by making the die slightly smaller so that the range of the servo is a larger proportion of the size of the die.

And with the weight close to the 6:

Moving the weight close to the 6 is certainly effective at making the die roll a 1. It rolls a 1 more than twice as often as any other number. It rolls a 1 about 40% of the time.

Subtlety

The biggest weakness is that if you shake the die, you can feel the weight inside rattling slightly. Anyone who has ever used a die before will immediately recognise that there is something going on inside, and at that point the jig is up.

The reason for the rattling is because the servo has some play in it.

Stavros suggested using springs to constrain the position of the weight. Alternatively I think you might get away with putting a strip inside on the 2 and/or 5 face, to press against the weight and stop it from rattling side-to-side. This would put more strain on the servo while moving the weight, but we don't care.

Practicality

To adjust the position of the weight, you need to hold the die on the cradle. But if you have the sleight of hand skills required to hold the die on the cradle without anyone noticing, then you might as well just carry 2 dice (one loaded and one not) and swap them as desired. There's really no need whatsoever for an adjustable loaded die. But I like the idea anyway.

(Apologies for the awful lighting in some of these pictures.)

You can see a crank handle on the right hand side of the front of the tablesaw. Cranking the handle raises the blade up and down: this feature works fine. Rotating the wheel that the handle runs inside tilts the top of the blade between 0° and 45° to the left. The tilting feature never worked satisfactorily. Between about 0° and 15° it was fine, but beyond that the gear teeth would start skipping unless you wedge the handle sideways, and it would take progressively more effort to get the blade to tilt the further you want it to go.

It recently got so bad that I was avoiding making 45° mitres because I knew it was too much trouble to tilt the blade over.

So I had a look at the mechanism. There is a curved toothed "rack" on the frame of the machine, and a pinion gear on the handle. I discovered that the gear on the handle was too small. Or, most likely, the designer deliberately spaced the gear teeth apart to stop them from binding when manufactured with sloppy tolerances, but went too far and now they don't mesh properly.

Upon taking the handle off, I discovered that not only were the teeth on the gear too small to mesh against the rack, but some of them had become partly broken due to the abuse.

So I designed a quick test gear in FreeCAD, 3d printed a test piece in PLA, checked the mesh of the teeth, and (after getting it the correct size on the 2nd attempt) superglued it to the existing gear just to check that it worked. Which it did, but the "draft" mode PLA teeth broke quite quickly.

Having discovered the correct shape of the gear, I then had the problem of how to attach my gear to the handle. I didn't want to 3d print an entire new handle. I settled for hacksawing the original gear off the end of the handle, facing off the handle on the lathe, and then gluing a new gear to the end of the handle with epoxy. Unfortunately I hadn't taken any pictures up to this point, but here's what I was left with after hacksawing the gear off:

My replacement is quite a lot bigger than the original!

The 2nd picture there probably best shows how the mechanism works. The gear is not keyed to the shaft. Turning the shaft raises/lowers the blade. Turning the gear engages with the rack, which causes the shaft to be dragged left/right which causes the blade to tilt. Obviously, if the gears don't mesh properly, then you can't move the shaft left/right so you can't tilt the blade.

For the real version of the part I modelled in some holes to help the epoxy mechanically hold the gear in place:

Printed this part in Prusament PC Blend, 100% solid (it's entirely made out of perimeters, no infill), and epoxied it to the handle:

Then it's just a case of installing it on the machine:

So far it's a great success. I wish I'd done this sooner. In the event that the printed polycarbonate gear breaks I will make a replacement out of aluminium using the CNC machine.

Here's a video demo of what I have so far. When you watch the video, just use your imagination and visualise the same sort of thing, except the electromagnets are really small and powerful, and there's maybe 64*64 of them, and they're switched automatically by a microcontroller instead of by hand.

The idea for this project came from the Regium KickStarter scam. Regium is an automatic chess board. I actually built an automatic chess board myself. Like most of the commercial equivalents, my board used a single electromagnet on an X/Y motion stage to move the pieces around. In contrast, Regium claimed to have a high-resolution grid of tiny electromagnets just underneath the playing surface. This would allow them to move the pieces around silently, to move multiple pieces at once, and to hold all of the pieces in place while you move the board around so they don't fall over. The only downside is it doesn't exist.

But I love the idea and I want to see how close it is to being possible, so I made a few electromagnets and connected them up in this 3*1 "grid".

Shortcomings

I wanted to make an actual grid for the proof of concept, even if only 2*2. But I did a handful of small experiments to decide how much wire to put in one electromagnet, and it turned out I only had enough wire for 3 of them. And I actually ran out before the 3rd one was done, so the middle one is smaller.

So in the absence of more wire, a 3*1 grid will have to do.

The electromagnets that I have made so far are both too large and too weak to make an effective chess board. The bobbins are 16mm diameter, and the housing puts the electromagnets 18mm apart. At 10v the 2 larger coils draw about 350mA and start to get warm if switched on for a while, so I wouldn't want to put any more current through them. They're barely strong enough to pull the steel ball around, let alone slide chess pieces across a board.

I did find that the smaller coil, drawing 660mA at 10v, is actually stronger than the larger coils, although it heats up even faster. Maybe the answer is to get some slightly thicker wire and run it at a higher current.

The core is made out of a mystery steel. I have no idea if it is good, bad, or about average for this purpose.

It may well be that it is not actually possible to make a working automatic chess board this way. You need to have either lots of turns, or high current, to create a strong magnetic field, and both of those place physical limits on how small you can make the electromagnets. It's quite possible that the achievable resolution is just not good enough.

Questions

I think this project would benefit from knowing the answers to some/all of the following questions. If you know the answers, or know someone who does, please email me: james@incoherency.co.uk.

1. How do we quantify how effective a candidate material would be to make the core out of? I want to be able to compare any 2 different materials, to find out which one is better & by how much.
2. What is the best material to make the core out of?
3. What is the best wire diameter? You can fit more windings in the same space if the diameter is lower, but you can't put as much current through it.
4. Short of adding more turns (makes it larger), or more current (needs thicker wire, which makes it larger), how can we make the electromagnets stronger?
5. How much of a difference does the diameter of the core make? We want it small to increase the resolution.
6. Does adding turns via more height actually help, or does it just serve to shift the centre of the magnetic field further from the surface?
7. Would making the core encompass the outside of the coil be advantageous, disadvantageous, or irrelevant? Note that we want the field to extend away from the surface, we aren't just looking for the highest possible density at points immediately close to the coil.
8. How high a frequency do we need to switch the coils on/off if we want the field to stay relatively constant?
9. Would adding a capacitor to each coil, to form a low-pass filter, help with multiplexing them?
10. What are some potential applications for this technology beyond an automatic chess board?

Commercial electromagnets tend to have the core surrounding the coil except for a small gap at the top, and they also tend not to be excessively tall.

(Pic from Adafruit. The coil is underneath the black seal; the core is the shiny part.)

However, commercial electromagnets are normally aiming to grab things that they contact. They would rather have a very concentrated field right next to the surface of the electromagnet, whereas we would rather have the field extend further away from the electromagnet, even at the expense of field strength immediately adjacent to it. So I'm cautious of imitating the common commercial designs because I think they may be counterproductive for our application.

How I made it

I 3d printed the end caps:

They have a 16mm outer diameter, and the centre is a tight fit on a 4mm core.

I chucked some 4mm mystery steel rod stock in the lathe, slid on a couple of end caps, taped down the starting end of the wire, and ran the lathe, using my fingers to guide the wire onto the core. Here's a clip of that process:

(Yeah, I know, the lathe is dirty and I should clean it).

Having wound the entire coil, I wrapped the windings in tape to stop them coming loose, and then cut the coil free of the rod stock with an angle grinder. I think if I were to make more of these I would cut the cores to length ahead-of-time, and come up with some alternative way of holding them in the lathe while winding the coil. There's too much risk of damaging the wire with the grinder otherwise.

Since I had no natural reference for the relative positions of the 2 end caps, the 3 coils ended up slightly different lengths, so when I 3d printed the housing I compensated for this in CAD:

You can see the left hole is shorter than the other 2, which are also not exactly equal. Having printed the housing, we have these parts:

I then dropped in each coil, soldered to copper wires at top and bottom, and wrapped it in sellotape to keep them in place, and to allow the ball to roll around more easily:

The bottoms of the coils all share a common ground. I would use a similar plan if I were making a "real" grid rather than a straight line. There would be a wire at the top for each column and a wire at the bottom for each row. To energise a single coil, you have to connect that coil's row (for example) to 10v, and its column to ground. Then the only coil that has a complete circuit is the one at the intersection of the selected row and column. The benefit of this is that as the grid size grows, you only need to be able to switch O(n) connections to address an arbitrary coil in an n*n grid.

Finally, I connected some buttons to the coils so that I can power them up independently:

Maglev chess

While talking about this project, Peter suggested that "maglev chess" would be a fun idea. You wouldn't even need it to be automated in any way, I think just the sensation of manually moving chess pieces around on top of a surface that makes them levitate would be fun. I don't know enough physics to know the best way to do it, but I think you'd want the centre of each square to repel the pieces, and the edges of the square to repel the pieces even more strongly, so that the only place for the piece to go is up in the air.

You also need to make sure that the pieces don't fall over, either by putting the magnets at the top so that gravity works for you (but then you need the repulsion to be much stronger), or by having multiple poles on the bottom so that the piece can't tip in any direction.

Chwazi:

(photo from la-matatena.com)

Well, I finally thought of a useful app to make for my Bangle.js watch. It's called "Choozi". Here's a demonstration:

The watch doesn't have a multitouch screen, and even if it did it would be a bit unwieldy asking multiple people all to put one finger on the watch screen. So instead it just distributes segments in a circle around the outside of the screen. The intention is that the group would be seated in a circle, and it would be clear which position corresponds to which person.

I am quite pleased with the animations. They are actually pushing the device pretty far. That's as fast as it can render those animations, there's no kind of delay or frame rate calculations at all, it's just drawing each frame one after the other as fast as it can.

The Bangle.js development process is slightly strange. You first write your app in the Espruino IDE (using Chromium because Firefox doesn't support Web Bluetooth) and test it out on the watch by uploading it to RAM via Bluetooth. Once you're happy with it, and you want to add your app to the menu so that you can use it again, or to the official Bangle.js apps loader so that other people can download it, you need to make a clone of the BangleApps repo, make a directory for your app, copy and paste your code out of the web IDE and into a source file in that directory, and then modify apps.json to include your app. And then use the Espruino Image Converter to turn your logo into an Espruino graphic suitable for use on the watch. Make sure you output as "Image String" instead of "Image Object", otherwise it won't work and won't tell you why.

I have submitted a pull request to get Choozi added to the official loader. If it's accepted then you should be able to install it on to your watch from there. In the mean time, if you want it, you can install it from my fork of BangleApps loader instead.

If you don't know what a neural network is, maybe start with the Wikipedia page.

Here's a diagram of the kind of network I was using (mostly a bit bigger than this, but not a lot bigger because training on CPU is slow):

Here's about the best I came up with, after about 3 hours of training, next to the training image (can you tell which is which?):

I had rather hoped for a bit better than this, although the generated image does have quite an interesting aesthetic in its own right.

Here's a video showing how the output improves as training proceeds:

More outputs

Here are some other interesting-looking outputs, not as good as the above. Unfortunately I didn't save the parameters used because I'm not a real scientist.

You can almost make out an outline of a face in a couple of them. If you squint.

These pictures remind me of the kind of nondescript, inoffensive artwork you sometimes find in hotels. It obviously looks like something, but it doesn't look like enough of anything that anybody could possibly be put off by it.

Some of the states from early on in the training, with less detail, are also reminiscent of old Ubuntu desktop wallpapers from around Ubuntu 10.10 to 13.10.

(left: early state from neural net of Lenna; right: Ubuntu 13.10 default wallpaper)

How

I used the AI::FANN perl module, which I think was last updated in 2009, so I imagine the state of the art is now substantially better (probably you'd use TensorFlow if you knew what you were doing?), but I found AI::FANN easy to get started with, and it entertained me for an evening.

The input image's (x,y) coordinates are scaled to fit in the interval [-1,1]. The (r,g,b) values are scaled to fit in [0,1]. (I tried various options and this seemed to work best, but... not an expert, don't know why).

We then create a training set consisting of (x,y) => (r,g,b) for every pixel of the input image, and initialise a neural network with 2 input nodes (x and y), 3 output nodes (r, g, and b), and some number of hidden nodes in some number of hidden layers that we can play with to see what difference it makes. I found I had best results with 3 hidden layers, each with fewer than 20 hidden nodes.

We then run Rprop training over and over again to make the neural network better approximate the function described by the training data, and periodically save the current state so that it can be turned into a video later.

If you want you can use the perl script I was playing with - but it's not really meant for general consumption.

And to turn a directory full of generated images into a video, I used:

$ ffmpeg -r 25 -i %04d.png -c:v libx264 -vf scale=800x800 out.mp4

Why

I just did this to see what would happen, but if it worked a bit better I can think of a couple of interesting applications.

Firstly, it might be an interesting way to lossily-compress images down to almost arbitrarily-small sizes, in a way that retains detail differently (if not necessarily "better") than traditional lossy compression algorithms.

Secondly, it can be used to upscale images in a way that might look more natural than normal interpolation algorithms, because the neural net (presumably) encodes some more abstract information about the shapes in the image, rather than the pixel values.

Potential improvements

I did try reducing the image to black-and-white to see if reducing the amount of work the network had to do would make it train more quickly, but it didn't seem to make a lot of difference. It generated the same kind of images, but in black-and-white.

Also, since the output images are clearly lacking so much detail of the input image, I tried downscaling the input image, on the basis that having fewer training inputs would make the training run more quickly. And it did help, but obviously that's not a general solution.

The neural net has to be quite small because I'm training it on the CPU and it is slow. It would be interesting to see if it would be much better with much larger networks trained on a GPU. I don't know if a larger network would actually help that much. I still think my network with 3*15*3 hidden nodes could do a much better approximation than what I'm seeing, so I think the real limitation is in the training method, which GPU speed would also presumably help with.

Finally, I think it would be interesting to try training it with a genetic algorithm instead of with backpropagation. You'd maintain a population of candidate weightings for the network, and at each generation you evaluate each candidate for fitness, and then breed the best handful together by repeatedly selecting weights from 2 "parents" to produce 1 "child", with a small amount of mutation (and maybe even some backpropogation training!), to go into the next generation's population.

I think I'm almost ready to bolt the box together and not open it again. I just need some way to transfer files between the SCAMP and my Linux PC, which would be either a removable CompactFlash card or a secondary serial port. There is already some hardware provision for both of these, so it might just be as simple as plugging a second 8250 into the serial card and connecting up the pins on its front panel.

Here are some pictures of the computer:

All of the electronics are mounted to the base, with a small panel at the back holding the power socket and at the front holding the power/reset buttons and VGA console. Then the wooden box can drop over the card cage and bolt down to it. The wooden parts are made of 15mm "Russian birch" plywood. I believe this is sold as "Baltic birch" in other parts of the world. It's great quality stuff, I like using it.

I'm quite pleased with the joints on the box:

I made all of these parts on the CNC machine, with a deliberate slight overhang on the joints, and then planed them flush by hand after gluing.

And here's a short video showing the current state:

(In the video the serial port output doesn't work at first, and I suggest that the VGA console might not have been ready. But on further reflection this can't be it, because the VGA console is already drawing a cursor. The output actually starts working after the point that the kernel re-initialises the UART, which suggests that when the bootloader tried to initialise the UART it didn't work. I don't know why that would happen. Possible problems ahead...)

Front panel

I was originally planning to put the clock and power/reset buttons on a dedicated card plugged into the backplane, but given that I only have 2 spare slots, I decided I'd rather not waste one. So power and reset buttons are on a panel mounted to the case, and the clock is super-glued to the backplane.

It might have been nice if I'd got 2 buttons that were both the same size.

I really like the effect of the power button lighting up when it's switched on. I'm also quite fond of the text engraving. I did this with a 2mm single-flute down-cutting end mill

The VGA console is attached to the back of the front panel with a 3d-printed bracket. It is extremely close to the power supply. You may not believe it from the photograph, but it actually doesn't touch - but it can do if you press on the front panel, so I have prevented it from shorting out by sticking some insulating tape to the power supply.

You can see I have very skilfully managed to stick the mounting screw out the back of the panel. Well, at least I didn't stick it out the front! The rest of the case is only held together with glued joints, but I decided to screw this one because I might need to take it off to gain better access to the stuff mounted on the back.

Because there's not a lot of room in this part of the case for the VGA console PCB, I went through a handful of 3d-printed iterations of the panel before cutting it out on the CNC machine.

Clock

Don't laugh: the clock is mounted to the backplane upside-down with super glue.

This is kind of hacky but it'll do for now. If it doesn't cause any problems then it'll probably stay like this permanently. I think the most likely problem is that I'll bash it while trying to drop the case on, and short one of the pins.

It is just a 1 MHz crystal oscillator. It seems like a great invention, I wonder why anyone ever uses a bare crystal.

Power supply

The power-supply is screwed to the bottom panel of the wooden case:

The AC power socket in the back has an integrated switch and fuse. Unfortunately the fuse holder is an unusual size and I couldn't find a fuse that would fit, and since UK power plugs have integrated fuses anyway, I decided it was fine (arguably best) to bypass the fuse in the power socket.

I'm not sure if the capacitor across the 5v output is doing any good. I don't expect it is doing any harm.

Wiring

We need to run a few wires from the backplane out to the front panel. Namely:

• Power & ground
• Reset
• TX & RX of serial console

It's reasonably tidy, and running all the wires down one side means I have plenty of room to work on it if I unscrew the front panel and cut off the cable ties - I can then bring the whole panel around to the side of the machine with a good amount of slack in the wires.

CompactFlash

I previously thought my original CompactFlash interface was going to be adequate, but after wiring up the new front panel, mounting the power supply inside the case, and moving the clock source to the middle of the backplane, it didn't work any more. Bummer. I'm not sure if it didn't like the new clock placement, or didn't like being so close to the power supply, or something else. Don't know. This is kind of the "dark magic" side of electronics for me.

So I made a 2nd revision of the CompactFlash interface, this time putting 74HC245 "bus transceivers" on the data lines between the card and the bus, as recommended by PickledDog.

It seems to be an improvement, in the sense that it now works with both of my CF cards where the previous one only ever worked with one, and it now works up to 1.2 MHz (although there is other instability at that speed).

But it didn't magically make the computer start working on its own, so there is obviously something else about the CompactFlash interface that is a bit sensitive. I found that moving it to the middle of the bus made a big improvement. I don't really know why. I wish I understood this stuff better. My mental model is that a conductor is at either 0v or 5v, and voltage changes propagate infinite distance instantaneously. It's frustrating that this is not a sufficient model for even my puny computer at my puny clock speed.

Other

SCAMP has been accepted into the Homebrew CPU ring. I think I'm expected to put the webring home/previous/next/random buttons on the project page, but since the project page is currently just a README.md on github, I've settled for adding a link to the Homebrew CPU ring in the "Resources" section.

I've been reading Starting FORTH, a FORTH tutorial from I think 1981. You can read the PDF online. FORTH is a fascinating language, but I still don't think I'm completely sold. FORTH code quickly gets completely unintelligible if you don't break it down into a large number of small "words", but the requirement to find a word to describe each part means you end up with lots of words that all sound the same but do slightly different things. I also found it to be slightly inconsistent with the Reverse Polish Notation. For example, : lets you define a new word, but it has to come before the word you're defining, because it operates on the input stream rather than the stack. But all words can do that: any word can take its argument from the input stream instead of the stack. You just have to develop a mental model in which it is clear where the arguments for any given word are going to come from, but I'm not there yet.

I read that to truly understand FORTH you need to implement your own, and implementing FORTH sounds like a much more compelling proposition than actually writing programs in it. So I think it's fairly likely I'll be making a FORTH system for SCAMP, and from there it's fairly likely that the edit-compile-test loop in FORTH will be substantially faster than in SLANG, so if I get on with it I may end up using FORTH for some of Advent of Code, which might be fun.

You can play with it here: SCAMP Emulator ».

Fun things for you to do with the emulator include playing Hamurabi, writing a "Hello, world" program in SLANG using kilo, finding new and interesting ways to cause kernel panics, and fixing whatever bug is causing half the files that should be in /src/ to go missing.

(If you actually do manage to cause a kernel panic, and it's not one of: deleting files from /proc/, filling up the disk, tinkering with kernel memory - then please email me and let's try and fix it!).

The disk image is not persistent, which means you lose all your work as soon as you close the tab, and there's also no way to upload or download files. Maybe that'll come later, maybe not. There's also no documentation, so... good luck.

This was actually my 2nd attempt at putting together a web emulator. The first time I gave up because I couldn't think of a quick and easy way to share the C code between the CLI emulator and the web-based version, because the CLI version's profiling and debugging hooks get in the way. This time I decided to just copy and paste the code and throw away the parts I don't want.

You might think that having 2 near-identical copies of the same code is distasteful: I agree. But I'd rather have an ugly program in the computer than an elegant one in my head. Perfect is the enemy of good, etc. Maybe one day I'll tidy it all up so that the common parts are shared. (Ha, good joke).

Emscripten

Setting up emscripten is much easier than you might expect. You can just copy and paste ~5 commands from the getting started guide, and then you're ready to go. After that, I didn't bother with the rest of the official guide, I instead followed a much shorter introduction from Mozilla.

I've been really impressed with emscripten. It seems that at every turn, if it's possible for emscripten to do some magic to make your life easier, then it does it. It even does a little bit of magic that doesn't seem possible. I'd like to use emscripten more often.

Xterm.js

Xterm.js is also really good, but its documentation leaves a lot to be desired. (Yeah, yeah, that's a bit rich, I know).

It has the kind of documentation that you get if you think comprehensive auto-generated descriptions of internal interfaces is preferable to short sections of typical example usage.

Apart from the auto-generated interface documentation, you're in good hands if you want 4000 words on Parser Hooks & Terminal Sequences, but if you want to know how to resize the terminal to 80x25, you have to read the source and figure it out yourself.

(Protip: term.resize(80,25). If this is documented then I certainly couldn't find it).

Apart from the documentation, Xterm.js is actually quite easy to use. Just make a <div id="terminal" />, let term = new Terminal(); term.open(document.getElementById('terminal'), hook onKey() to find out about user input, and term.write(...) to provide output. It pretty much handles being a terminal emulator exactly the way you'd expect.

Building the emulator

Emulating the CPU itself is mostly simple. I just simulate all the changes to the control signals on a negative clock edge, and then a positive clock edge, and then repeat. The annoying parts are interfacing with the terminal, populating the disk contents, and populating the boot ROM and microcode ROM contents.

To populate the boot ROM and microcode ROM, I wrote a Perl script to convert the ROM files into C source files that contain their contents. I initially tried to use the same technique for the disk image, but at 32 megabytes it was too big to compile on my laptop and emcc ran out of memory trying to compile it.

I subsequently learnt that emcc has an option, --preload-file, that allows you to include existing files inside the emscripten environment. This turned out to be perfect: just compile with --preload-file os.disk and then read the disk contents from os.disk at runtime using normal C filesystem interaction. This is really incredible technology. I expected emscripten to compile C code to WebAssembly, I didn't expect it to provide a simulated operating system!

My complete command line is:

emcc -o scamp.js scamp.c ucode.c rom.c --preload-file os.disk -s WASM=1 -O3 -s NO_EXIT_RUNTIME=1 -s EXPORTED_RUNTIME_METHODS=['ccall'] -s ALLOW_MEMORY_GROWTH=1

I think NO_EXIT_RUNTIME means your program maintains its state even after main() returns. Putting ccall in EXPORTED_RUNTIME_METHODS lets you call C functions from JavaScript, and ALLOW_MEMORY_GROWTH is required because I want to allocate 32 megabytes of memory to store the disk image. (I actually load the disk image with mmap() because that's how I did it for the CLI emulator, and amazingly this "just works" thanks to yet more incredible emscripten technology).

Part of the point of using emscripten rather than writing the emulator in Javascript is that I don't trust Javascript to be sufficiently performant to run the CPU at 1 MHz. To that end, I want to keep Javascript out of the loop as much as possible, so my "please run a clock cycle" function takes an argument that says how many clock cycles to run, and that way we don't need to initiate 1 million function calls per second from Javascript:

EMSCRIPTEN_KEEPALIVE
char *tick(int N, char *input) {

(EMSCRIPTEN_KEEPALIVE prevents the compiler from optimising the function out on the basis that it is never called - we need it because we'll be calling it from Javascript).

The idea of tick() is that console input characters are passed as an argument, and console output is returned. This is kind of weird and unpleasant, but it does the trick. Calling this function from Javascript looks like:

let pending_output = ccall('tick', 'string', ['number', 'string'], [10000, pending_input]);

The ccall() arguments are:

• the name of the function to call (tick)
• the return type of the function (some more emscripten magic turns the char* into a string for javascript - incredible)
• the types of the arguments (even more magic turns the javascript strings into char* for C!)
• the values for the arguments (run for 10000 clock cycles, append pending_input to the UART input buffer)

Having spent most of this year writing low-level code for SCAMP, where the computer can't even work out if you've passed the correct number of arguments to a function, it blows my mind that emscripten can just magically turn Javascript types into the appropriate C types for the specific function you're calling, and it all just works. Very impressive stuff.

To get IO to/from Xterm.js, keypresses and text output are sent between the main web page and the web worker using postMessage().

Web stuff

Charlie complained that the page took a long time to load over a mobile connection. This was almost entirely down to having to load the 32 megabyte disk image, but since the disk is almost completely empty, it seems a bit wasteful. I asked nginx to compress it:

location /scamp/scamp.data {
    gzip on;
    gzip_types application/octet-stream;
}

And that solved the problem: it now only transfers 362 kilobytes for the disk image, almost 100x improvement.

kilo originally used Ctrl-S to save the file, but this collides with software flow control, which means I can't save anything if I'm using the real hardware via a USB serial cable in screen. So I changed it to Ctrl-W to write the file. Unfortunately Ctrl-W collides with "please close my tab and don't let the page intercept the keypress" in Firefox, which means it's impossible to save anything in the web-based emulator. So now I've changed it to Ctrl-O, which matches nano and stands for "output". I can't wait to find out what sort of UI Ctrl-O is going to interact poorly with.

Here's a short demo of inaccurate play:

It's actually pretty easy to win the game with the highest praise almost every time if you know how it works. Perhaps I'll write up "How to win at Hamurabi" at some point.

If you want, you can play the game online in a handful of places. Probably the most authentic experience is the one on "playold.games". It appears to be a DOS port of the game running in the browser in a javascript DOSbox. Or if you really want, you can clone the SCAMP repo, work out how to build everything, and run hamurabi in the SCAMP emulator. Email me if you can't figure it out.

Porting Hamurabi

The original game was apparently written in FOCAL, although at that time it was called "King of Sumeria" or "The Sumer Game". Its roots can be traced back further, however, to The Sumerian Game from 1964, so it's not entirely obvious to me where Hamurabi can be said to have started. I'd be interested to look at the FOCAL source code, but I couldn't find it. Fortunately, the game has had many ports over the years.

The earliest source code I found was this low-res BASIC listing from the book BASIC Computer Games published in 1978, so that's what I ported to SCAMP, bugs and all. Of course, that's not to say I didn't introduce extra bugs of my own.

The full source of my port is in sys/hamurabi.sl in the SCAMP git repo.

Arithmetic

I was surprised to find that such an old game appears to make extensive use of (presumably) floating point arithmetic, e.g.:

552 D=P-C: IF D>.45*P THEN 560

I don't yet have any fractional number support on either my CPU or programming language, so I converted these calculations into pure integer form for SLANG:

if (mul(D,100) > mul(45,P)) {

(The CPU doesn't natively support multiplication, so neither does the programming language - I considered implementing infix multiplication as syntactic sugar, but decided the explicit function call is fun and quirky. To be honest it was touch and go whether the language would even have a non-buggy magnitude comparison operator...)

Input

At first reading I thought the game systematically omitted question marks on questions:

320 PRINT "HOW MANY ACRES DO YOU WISH TO BUY";
321 INPUT Q: IF Q<0 THEN 850

This presents a real dilemma: I don't want my port to deviate from historical accuracy, but I also don't want to ask questions without question marks.

Fortunately, I later came across a screenshot from the game which clearly shows question marks on questions:

(From Wikipedia)

So I have surmised that the INPUT statement in BASIC inserts its own question mark, and also that the trailing semi-colon after a PRINT statement suppresses a trailing end-line character.

That said, the screenshot clearly shows "O MIGHTY MASTER" wording after "THINK AGAIN", which the BASIC listing definitely doesn't have, so who knows? Maybe the BASIC version actually didn't have question marks.

Comments

The BASIC source code is very sparsely commented, and the comments don't even address my biggest questions. I would have appreciated comments explaining how the calculations are meant to work, and what the single-letter variable names are meant to represent. Instead, these things have to be derived by working backwards from the impact they have on the text output. But at least the immigration calculation has this:

532 REM *** LET'S HAVE SOME BABIES
533 I=INT(C*(20*A+S)/P/100+1)

Control flow

IF statements in this dialect of BASIC appear to function more like a conditional jump than a normal if statement, in that they take a line number as a branch target, rather than a block of code to conditionally execute. This makes the control flow quite hard to follow. The language also doesn't appear to support named functions, although it thankfully does have a RETURN statement, so some form of proto-subroutine is possible.

Example:

410 PRINT "HOW MANY BUSHELS DO YOU WISH TO FEED YOUR PEOPLE";
411 INPUT Q
412 IF Q<0 THEN 850
418 REM *** TRYING TO USE MORE GRAIN THAN IS IN SILOS?
420 IF Q<=S THEN 430
421 GOSUB 710
422 GOTO 410
430 S=S-Q: C=1: PRINT
...
710 PRINT "HAMURABI:  THINK AGAIN. YOU HAVE ONLY"
711 PRINT S;"BUSHELS OF GRAIN.  NOW THEN,"
712 RETURN

Also note that there is nothing at line 410 to mark it as the start of a loop body, or at 710 to draw attention to the fact that this is the entry point of a subroutine. You just have to know. In my opinion, the SLANG port is substantially easier to follow, despite my efforts to mimic the original coding style.

while (1) {
    printf("HOW MANY BUSHELS DO YOU WISH TO FEED YOUR PEOPLE",0);
    Q=input();
    # *** TRYING TO USE MORE GRAIN THAN IS IN SILOS?
    if (Q <= S) break;
    thinkgrain();
};
S=S-Q; C=1;
printf("\n",0);
...
thinkgrain = func() {
    printf("HAMURABI:  THINK AGAIN. YOU HAVE ONLY\n",0);
    printf("%d BUSHELS OF GRAIN.  NOW THEN,\n",[S]);
};

(The IF Q<0 THEN 850 test is moved into the input() function because it's the same on every input - if you try to input a negative number the game tells you "HAMURABI:  I CANNOT DO WHAT YOU WISH. GET YOURSELF ANOTHER STEWARD!!!!!" and you lose)

Random numbers

Random numbers are an important part of the game. They create uncertainty about the future which mimics the real-life conditions that the game is trying to simulate. I don't know how good the BASIC random number generator is, but I know that the SCAMP one is awful. This is the first program I've written for SCAMP that needed random numbers, so I wrote this:

var randstate = 0x5a7f;
var rand = func() {
    randstate = mul(randstate, 17) + 0x2e7;
    return randstate;
};

The seed is the same every time, so it is easy to learn what the land prices and harvest yields are going to be, when there will be a plague, and so on, but it's a start. The form of the RNG (X_n+1 = (a X_n + c) mod m) is a linear congruential generator, but the constants (a=17, c=0x2e7, and implicitly m=0xffff), are chosen by guessing instead of for their good properties. I definitely want a better RNG for SCAMP eventually, but this is adequate to make the game work for now.

I've been thinking a bit about how I might seed the RNG so that it isn't the same every time. It would be quite easy to have the kernel count loop iterations between booting up and the first keyboard input, maybe that's the best way to seed it.

Other SCAMP stuff

Case

I have made a start on the wooden case. Here's some photos of the card cage inside the case, with no cards installed:

I still have some sanding to do to tidy up the joints, but it is roughly what I wanted. Removing the domed nuts at the top allows most of the case to lift off, allowing access to the back and sides of the card cage. The card cage is more permanently bolted to the base piece, which also holds the power socket and power supply.

Originally I was going to put the power switch, reset switch, and clock on a PCB in the card cage, but given that that would only leave 1 spare slot to experiment with, I think I'd rather put them on a separate panel mounted to the bottom of the case, and bodge-wired to the backplane, to leave a 2nd slot free for expansion.

Clock

I experimented with the RC2014 clock card to find out how high I could clock SCAMP without anything going wrong. I was disappointed to find that the limit is about 1 MHz. The next step up available on the RC2014 clock is 1.2 MHz, and at that speed I couldn't get the CompactFlash card to work. I did attempt Bill Shen's suggestion of putting 100 Ohm resistors on the data lines and an RC filter on IORD, but it didn't appear to make any difference. So I've given up for now and I'm just going to tolerate running it at 1 MHz. It would be nice to go faster, but I can always revisit it at a later date. Getting it finished is more important. Rory suggested adding a pull-up/-down resistor to every line on the bus to help the levels transition more quickly. I haven't yet tried this.

Apart from the CompactFlash card, everything else seems to work up to at least 2.5 MHz, and after that point text output stops working, and I can tell from looking at the lights that the CPU isn't working properly either. But if I can fix the CompactFlash problem then 2.5 MHz should be within easy reach. I then don't know which part is making it fall off the rails above 2.5 MHz, and don't know exactly how I'd go about debugging it. Maybe I'd need to attach a pretty wide logic analyser to the bus and see if I can work out what is happening?

Other games

Tetris would be good, to complete the nand2tetris theme. Snake might also work, although it would have to be played on the 80x25 text terminal. I think Colossal Cave Adventure is an almost mandatory port. And a Z-machine implementation would open up a great wealth of other text adventures for free.

If you read this review and enjoy it but think you now no longer have anything to gain from reading the book, then know that precisely the opposite is the case! I am barely scratching the surface of the material the book covers. It is a tome of great wisdom.

Adam Savage

If you only know Adam Savage from MythBusters, then you may be under the impression (as I used to be) that his expertise is primarily as a TV presenter, and only secondarily as any kind of builder of things. This couldn't be further from the truth. In the book, he describes himself thus:

I'm a generalist in my creative output and a wannabe polymath in how I organize my life.

My talent lies not in my mastery of individual skills, at which I'm almost universally mediocre, but rather in the combination of those skills into a toolbox of problem solving that serves me in every area of my life.

(And if this sounds interesting to you, you may also enjoy Range: Why Generalists Triumph in a Specialized World).

There are probably hundreds to thousands of hours of Adam Savage, talking and making things, available on his Tested YouTube channel. If you want to watch some but don't know where to start, I might recommend his machine vise build or this segment on imposter syndrome.

The "Maker" label

The book mentions a common aversion to the "maker" label: that of considering oneself not good enough to call oneself a maker. He brings up an example of someone he spoke to at a Maker Faire who said "I don't make, I code":

The list of exceptions people invent to place themselves on the outside of the club of makers is long and, to me, totally infuriating. Because the people who do that to themselves—or more likely, the people who TELL them that—are flat wrong.

And goes on (convincingly) to argue that any act of creation is an act of making, no matter what kind of thing you're making.

There is another take on the aversion to "maker", not directly addressed in the book: some might be wary of adopting the label out of a fear that "maker" is just a word for a substandard practitioner of a more specific craft. But I don't think that's quite right. Anyone who makes anything is a maker. The perception of "makers" sometimes doing subpar work is more down to the aggressive sharing in the maker community. You see more early work from those who describe themselves as makers than from those who follow more formal paths into craftsmanship. This should be encouraged, not squashed.

Philosophy

The book contains a lot of good insights on the philosophy of making, and why we do it. And more importantly, why we sometimes don't do it, instead running into barriers that we've erected inside our own heads. I was hooked from the introduction alone. If you read the book and the introduction doesn't excite you, then you probably needn't bother reading the rest. But it probably will.

Now I have quotations to present without comment.

On culture:

Whenever we're driven to reach out and create something from nothing, whether it's something physical like a chair, or more temporal and ethereal, like a poem, we're contributing something of ourselves to the world. We're taking our experience and filtering it through our words or our hands, or our voices or our bodies, and we're putting something in the culture that didn't exist before. In fact, we're not putting what we make into the culture, what we make IS the culture.

On permission:

[This book] is also a permission slip. The permission slip is from me, to you. It says you have permission to grab hold of the things you're interested in, that fascinate you, and to dive deeper into them to see where they lead you. You might not need that permission. If that's the case, good for you! Go forth and do awesome things. But I have needed that permission many times in my life. And whenever I found it, it helped me uncover secrets about myself and about the world I live in. It made me better as a man, as a maker, and as a human being.

On vulnerability:

As the things we make give us power and insight, at the same time they also render us vulnerable. Our obsessions can teach us about who we are, and who we want to be, but they can also expose us. They can expose our weirdness and our insecurities, our ignorances and our deficiencies.

On obsession:

We can't even countenance the idea that someone could be obsessed with something and be of sound mind. That is a shame, because when it comes to creativity, when it comes to making things, when it comes to success at anything, obsession is often the seed of real excellence.

On bullshitting:

Embrace the noun (Maker, Painter, Writer, Designer) by sharing with the world evidence that you've been living the verb (making, painting, writing, designing). Just don't be a bore, and definitely don't believe your own bullshit. Believe me everyone can tell the difference between someone who just talks the talk and someone who can walk the walk.

On sharing:

The book covers a time that Adam was trying to find work in Manhattan special effects companies, and found the staff to be insular, unwilling to share, as if they considered the sharing of their knowledge to be a threat to their livelihood. This was in stark contrast to the working environment he later found in San Francisco, where not only were the staff much more willing to freely share knowledge with one another, but the company owner (Jamie Hyneman, cohost on MythBusters) gave the staff free usage of the workshop during their down time to work on whatever they wanted.

It's up to you to decide what you're going to do with all the knowledge you accumulate. Are you going to hide it? Are you going to pretend that you alighted upon all your insights by divine providence? Or are you going to share what you have learned? Are you going to open yourself up to the people in your orbit and show them who you are, what you love, what you've made, what you know, who's helped you, and what you plan to do with all this to make the world a better place to live?

Related (but only obliquely covered in the book): telling others about your work increases your luck surface area, so you should do it even if you don't care about improving anyone else's lot.

Productivity

Adam writes lists to keep track of things. Todo lists, parts lists, shopping lists, and so on. Initially he would simply cross something off the list once he was done with it, but he found checkboxes to be superior. With the checkbox system, you draw an empty square next to each item of the list, and colour in the square when that item is done. This leaves the writing still legible, and also gives the opportunity to leave a square half-coloured-in when the job is half done.

The elegance and effectiveness of this planning system floored me, particularly when it came to evaluating the status of a project the further along it went. The value of a list is that it frees you up to think more creatively, by defining a project's scope and scale for you on the page, so your brain doesn't have to hold on to so much information. The beauty of a checkbox is that it does the same thing with regard to progress, allowing you to monitor the status of your project, without having to mentally keep track of everything.

Adam's workshop philosophy is "see everything, reach everything".

On drawers:

Let me tell you my philosophy about drawers: F*CK DRAWERS! Drawers are where stuff goes to die. Drawers lure you into a false sense of security by "helping" you put things away and making the shop look "cleaner". But really, by putting stuff out of sight, they remove it from your visual field.

He refers to the "see everything" half of this phrase as the "visual cacophony". The "reach everything" part is "first-order retrievability". He tells a story of having earnt some money and bought himself an expensive toolbox set, only to find that it didn't suit his style of working:

I gave the Kennedy stack a good four years to prove itself, but eventually, as a natural enemy of the visual cacophony, I had to get rid of it. In its place I built a rolling, five-runged , five-foot-tall ladder rack with twenty holes drilled into each rung big enough to accommodate the handles of every tool I'd kept in the Kennedy stack. I literally turned the storage drawers inside out. Each tool has a place that's fairly obvious, they are all in sight, and I can get to any of them five times faster than it took me just to figure out which drawer they used to be in.

On sweeping up every day:

Sweeping up every day and putting away my tools is a conversation between present me and future me. It is present me acknowledging that future me always likes to keep the momentum of a project going, and that having to look for a tool or walk across the shop to get one in the middle of a critical phase can slow down the creative process enough to be an existential threat to the project as a whole.

Prescription

Read it for the wisdom on maker philosophy. Also enjoy the discussions on productivity, workshop organisation, safe and efficient tool usage, snippets of Adam Savage's life story, and the surprisingly long section on all the different types of glue. And tolerate the distractions into film buffery and cosplay.

Here's a video I recorded showing the computer booting up, and writing, compiling, and running a short program:

I would like to share more stuff via video, but I'm not very good at it, and still feel silly while talking to the camera on my own.

Performance

It previously took about 19 seconds to get from reset to the shell, which is now down to 6 seconds, mainly because of the CompactFlash I/O improvements in the kernel. I wasn't specifically looking to optimise the boot time. 19 seconds is fine. A faster boot is just a happy side effect.

I/O

I have finally got around to rewriting the CompactFlash I/O inner loops in assembly language.

It's probably beyond obvious to real assembly language programmers, but I found that unrolling the loop was a big improvement. My first attempt was more like:

loop:
    in x, CFDATAREG # 5 cycles (read data from device)
    ld (r3++), x    # 8 cycles (write to buffer)
    dec r0          # 5 cycles (loop overhead)
    jnz loop        # 4 cycles (loop overhead)

You can see that it only spends 13 out of 22 cycles (59%) on instructions that are actually copying data. The rest is loop overhead. With the loop unrolled, that's now 104 out of 113 cycles (92%). We're copying 256 words each time we read a block from the card, so the total cycle count has reduced from about 5600 to about 3600 (-35%), just by unrolling the loop into groups of 8.

I found that the memcpy() on every disk I/O operation was also quite a significant bottleneck. That also benefited from loop unrolling, using the Duff's device trick of jumping part way into a multiple-of-N loop to copy non-multiple-of-N blocks of data. It's not particularly easy to read, but you can read my memcpy() implementation here. I found calculating the jump target too inconvenient, so I did it by indexing into an array of loop offsets instead. It ain't stupid if it works.

Text editor

The text editor was originally grabbing characters from the console via the kernel, using the read() call on the input file descriptor. This worked OK, except arrow keys didn't work. Arrow keys work by sending an escape character, followed by a "[", followed by a letter to indicate which arrow it is. Left arrow is the 3-character sequence "^[ [ D".

The reason arrow keys didn't work is that with the serial line running at 115200 baud, and the CPU at 1 MHz, we only get about 86 clock cycles to process each incoming character before another one is here. The overhead of grabbing each character via the kernel was just way too high. I tried slowing the serial line down to 9600 baud, but it still wasn't enough to handle arrow keys.

Even aside from that, I found the text editor almost unusably slow. It redraws parts of the display on every keypress and this was taking too long. I profiled this in the emulator and found that it was spending about 480,000 cycles between accepting a character of input and being ready to accept the next character. At 1 MHz, that translates to 480 ms, which means if you type any faster than 25 wpm, it's going to start dropping characters. I like to type at more like 120 wpm, so that's no bueno.

Most of the 480,000 cycles were spent on providing output via the kernel, rather than on calculating anything interesting.

So to solve these problems, I rewrote the text editor's text input and output code in assembly language, inside the text editor. It now interacts with the UART directly instead of using the kernel. This isn't ideal, because for example if I ever gain support for multiple console devices, the text editor will need to handle that separately instead of getting it for free via the kernel. But that seems unlikely anyway.

The editor now takes only 78,000 cycles to process an input character, which means it's good for about 150 wpm, which I was initially delighted with. However I eventually realised that this gets worse at the end of the line. It redraws the entire line every time the line changes, so when there are more characters it takes longer. At the end of the line this rises to about 213,000 cycles, or 56 wpm, which is still a bit on the slow side. So there's probably more work to do here. Probably I just rewrite the line-drawing code in assembly language.

Compiler

From watching the video, you might be surprised that it takes about a minute to compile a single-line program. This is probably my next avenue for optimisation, because compiling programs is one of the things I definitely want to be able to do with this computer. (My ultimate goal is to use this computer to solve as much as possible of this year's Advent of Code).

I'm not completely sure what it is wasting so much time on. Originally, compiling programs was slow because it re-compiled the entire standard library every time. But the library is now cached as an (approximation of an) object file that can just be dumped at the start of the generated binary. The pattern on the lights suggests that compilation is very I/O bound. I need to profile it and find out.

CompactFlash PCB

I have now got a PCB to hold the CompactFlash card:

The long wire that's not connected to anything is meant to indicate the internal card's "busy" status with an LED on the front, but I decided to rip off the pad that it was soldered to, so it doesn't work at the moment.

And it contains provision for a removable "external" card which can be inserted through the front panel. There is no software support for this yet (and in fact I may have done something wrong on the PCB, because with a CompactFlash card plugged in to the 2nd slot the computer won't even boot correctly; need to investigate).

Serial console

I'm currently using the RC2014 VGA Serial Terminal to connect SCAMP to a keyboard and monitor, but this isn't completely ideal because a.) I'd like to put this back in my RC2014, and b.) the pins aren't labelled, so it is quite tricky to make sure I am plugging everything into the correct ones.

The creator of this board, Marco Maccaferri, also designed a similar board that is a standalone VGA serial terminal, rather than plugging into the RC2014 backplane. Unfortunately he no longer offers kits for sale, but they are available from a website called connect.gi. I found the seller to be friendly and helpful.

I have received my standalone VGA serial terminal, but I decided to solder the 3.3v regulator (not pictured) on backwards so now it doesn't work. I then broke one of the legs while trying to desolder it, but hopefully once my replacement regulator arrives I will be able to switch to the standalone board.

Power supply

I bought this power supply ages ago but hadn't got around to testing it until this week:

It is a 5v 10A DC power supply. It turns out that 10 amps is about 9.5 amps more than I need, but at least it's futureproof.

One strange quirk of SCAMP is that it doesn't work properly at 5.0v, it needs more like 4.8v. I previously believed that this was just down to miscalibration of the voltage display on my bench power supply, but now I believe it is because I have not made any effort to meet the spec required for interfacing with the CompactFlash card. It seems to get stuck waiting for the card while trying to read the kernel off the disk if running at 5.0v. But for the time being, I don't care what the problem is. I just run it at 4.8v and it's fine.

The power supply has a trimpot on it to let you adjust the output voltage, but I found that even at the lowest setting it was barely below 5.0v, and the computer wouldn't boot. The trimpot was variable from 0 Ohms to 1 kOhm, and I found that the lowest output voltage was achieved with the trimpot set to the highest resistance. So the solution is clear: I just need to replace the trimpot with one that goes higher.

I replaced the 1 kOhm trimpot with a 2 kOhm trimpot that I already had in stock, and was able to set it to 4.8v and the computer boots! Great success.

Don't tell anyone I'm not qualified service personnel.

Bug fixes

One of the bugs last time was that the text output from the bootloader and kernel startup was missing a load of characters, because they tried to output bytes faster than the serial line could take them. I've now made all of the serial output code check the UART status before writing anything, so that problem is gone.

Another bug was that typing a space would sometimes insert a nul character instead of a space. I do not know what was causing this problem. It has gone away now that I'm not stuffing bytes into the UART faster than it can transmit them, so maybe I was running into a strange edge case of the UART? Not sure.

Another bug-in-waiting was that my "32 MB" CompactFlash card only has 62,270 blocks instead of 65,536. The SCAMP filesystem assumes that it always has 65,536 blocks to play with, so this is no good, and would eventually lead to block contents disappearing into a black hole. My current "solution" to this problem is to mark the missing blocks as "used" so that they can never be allocated for use. This isn't an ideal mechanism because it makes resizing the filesystem non-trivial (there is nothing to indicate which blocks are actually used and which are just missing). But it'll do for now.

Next steps

I have designed an initial CAD model for the case I'd like to build:

And ordered some fancy "Russian birch" plywood to make it out of. Unfortunately I failed to notice when ordering that the plywood has a lead time of about 25 days, so I won't be receiving that any time soon.

I still need to build a clock circuit. I have a bit of a mental ugh field around this. I'm not sure why. I think there is a bit of a dependency loop, because I don't know how to best design the clock circuit before I know what frequency it needs to run at, and I won't know what frequency the CPU can cope with until I have a better clock.

I would like to do some profiling of the compiler, assembler, and compiler driver, to work out why it is taking nearly 60 seconds to compile a 1-line program. I'm sure a lot of it is overhead that will scale sub-linearly with program size, but it definitely feels like this should be faster.

I have found that the VGA serial terminal does not seem to handle the page up/down keys, at all. It's not just that my text editor doesn't understand the escape sequences: it doesn't receive any escape sequences! If I can't work out how to get the VGA serial terminal to send them, then I'll probably just provide Ctrl-U and Ctrl-D as synonyms for page up/down.

I have also found that the VGA serial terminal sends code 127 for the "DEL" key, whereas gnome-terminal (or Linux, or something else in the pipeline) sends 127 for the backspace key. That's annoying. Before discovering that, I had just supported every observed variation of codes for each special key, so that it supports both types of terminal transparently. Unfortunately I am now in some danger of reinventing termcap, which I would like to avoid.

Since I no longer need the Arduino code to sync anything to the clock signal, I'm free to use the CTC to generate it, which can give a much higher frequency. I used TimerOne to generate a 1 MHz square wave (which is the maximum as far as I can tell), and the SCAMP hardware still appears to function correctly, which is great news.

Here's a recording of a session running on the real hardware (it's 6 minutes long so maybe you don't want to watch it all):

Performance

I know it seems quite slow, but consider that previously it took about 30 minutes just to boot up to the shell, and now it's almost usable interactively. Things are moving in the right direction.

There is potentially scope to clock it even higher, and there are definitely software improvements to be made. For example the kernel's CompactFlash reading/writing code is all written in SLANG at the moment, which produces extremely unoptimised code. Eventually I'll rewrite the inner loops in assembly language.

Obvious problems

The first obvious problem is the garbled text at the start. This is text output from the bootloader and kernel, and the reason it is garbled is because the CPU is now running fast enough that it tries to stuff characters into the UART faster than they can be transmitted! I didn't expect to run into this problem so early, but it should be straightforward to fix, either by adding a bunch of slownop in the worst case, or preferably by polling the UART status.

The second problem is that the space bar doesn't seem to work properly. I can not explain this. I didn't observe any flipped bits in any other keypresses, so it is hard for me to imagine what is going on. Sometimes pressing the space bar puts nothing on the screen, but invisibly terminates your input command line. It acts as if the space bar sometimes inserts a 0x00 instead of 0x20. Not sure what's happening, but I look forward to debugging it.

Next steps

The immediate next step is to turn this mess of wires into a PCB:

I'm very pleased to be at the point where the UART and the CompactFlash card are both working correctly. I have found interfacing with the peripherals to be the most complicated part of building the computer, because they are rather inscrutable black boxes that are hard to faithfully recreate in the emulator.

I think I might provide for 2 CompactFlash cards on the PCB. One would be purely internal and hold the kernel and root filesystem, and the other would be removable through the front panel, as a convenient way to transfer files between the SCAMP and a modern computer. The kernel doesn't yet support more than 1 block device, and I'm not exactly sure how I would support it. It might be simpler to have a "userspace" program responsible for interacting with the removable device, rather than mounting it on the filesystem.

It's probably about time that I looked at building a proper clock circuit. I'm curious to know how fast it can be clocked before it gets unstable.

And I still need to work on the wooden case that I keep talking about, fit a proper power supply inside it, and make a PCB for the clock, and the power & reset switches.

Demo

Here's a quick-and-dirty Matter.js demo of the simplest mechanism I could think of:

If it runs out of energy you can refresh the page or give it a kick with your mouse.

The pendulum on the left is a "normal" pendulum. The one on the right has a linkage that makes the effective pivot point of the pendulum swing left and right with the weight. This has the effect of flattening out the path of the weight, which approximates an arc of greater radius.

I had to "cheat" in the demo because I couldn't work out how to use Matter.js constraints to keep the angles of the short and long arm synchronised. The pivot at the top is actually static as far as Matter.js is concerned, and my code manually updates its position. This is why the green line has some slight deviations.

How it could work in real life

I wasn't clever enough to include a representation of the actual linkage in the animation, but I envisage something like this, where there is a sprocket at each end of the short arm, with a chain that keeps the angles synchronised:

The sprocket at "pivot 1" is fixed to the frame of the apparatus, and the sprocket at "pivot 2" is fixed to the long arm, so that as the long arm rotates, "pivot 2" pulls itself around the chain, which rotates the short arm around "pivot 1" because the sprocket at "pivot 1" can't move.

If you spend some time thinking about it you might undertand it. Instead of a chain and sprockets we could just as easily have a gear train with an odd number of gears, or a pair of pulleys with a string fixed at opposite sides, or possibly something even better.

Why it's good

I have a vague idea that I'd like to build a mechanical clock at some point. I'm kind of thinking a floor-standing clock of grandfather clock scale, but with less fine woodworking, and more exposed workings of unusual design.

I think it would be fun if the clock ticked really slowly and purposefully, with the second hand jumping several seconds with every tick. To achieve this with a standard pendulum, the clock would need to be very tall indeed. If the clock ticks twice per period of the pendulum (once for each direction), then to get a tick every 5 seconds you would need a 10-second period, which implies a pendulum that is 25 metres tall, which is clearly absurd.

Why it's bad

It is probably very difficult to build the two-armed-linkage with sufficiently-low friction that it would actually work in a clock. I don't know how much energy an escapement can typically impart into a pendulum with each tick, but it is presumably low, otherwise it would compromise the pendulum's regularity (maybe?).

More importantly, the fact that the two-armed-linkage pendulum does not swing through a true circular path means that its period is not truly constant. The mean radius decreases as the weight swings higher, which presumably means the period decreases as the weight swings higher. To get a constant period out of this pendulum, you'd need to ensure that it swings to the same height on every stroke, and if you can do that then you don't need a pendulum in the first place.

My thinking is that the period won't vary too much throughout the useful range of the pendulum, and it doesn't matter if the clock drifts a bit anyway. It would be good to do some calculations to quantify how the period changes with the swinging height. But I can't immediately see how to do that.

Parameters

A normal ideal (spherical cow, frictionless vacuum, etc.) pendulum only has 1 parameter: the length. Or, more precisely, the radius of the arc that the centre of mass swings through.

Our two-armed-linkage pendulum has three interesting parameters: the length of the long arm, the ratio of the length of the short arm to the long arm, and the gear ratio of the sprockets.

As the short arm length tends towards 0, the motion of the two-armed-linkage pendulum tends towards that of the classical pendulum. As the short arm length tends towards that of the long arm, the "arc" tends towards a straight line, where gravity will no longer have any effect on the weight. Where the "short arm" is longer than the "long arm", I think the arc is inverted and gravity will pull the weight towards whichever side it is already on, rather than always pulling towards the centre.

It's not immediately clear to me what happens as the gear ratio changes. My demo and sketch assume a 1:1 ratio, but it is possible that there is a better choice which would more faithfully approximate a circular arc.

Well wonder no longer!

(not pictured: there are springs on the sandpaper "fingers" so that they apply force inwards)

Simply chuck up the little shaft to your cordless drill, stick the bar you need to sand through the large hole, then spin up the drill and the little pieces of sandpaper will polish your bar.

Drawbacks

Currently this is just a proof-of-concept and there are some clear drawbacks with the design:

• the casing is enormous compared to the size of bar you can sand
• the springs that press on the sandpaper are too weak
• the pieces of sandpaper are too small
• it is difficult and inconvenient to load the bar because you have to hold all 3 fingers out of the way simultaneously
• the gear ratio is too extreme, so the sandpaper turns too slowly
• making the small shaft and pinning it to the small gear is time-consuming, I should just have a hex hole in the gear driven by an allen key tool in the drill, or similar

But I think the concept is proven, and if I can be bothered I might design a better version and print it in a better material than the PLA used here. I've been making mechanical parts with Prusament PC Blend recently, which was released without much fanfare and I haven't seen very many people talking about, but which I think is a brilliant material. It appears to be much stronger than PETG and hardly any more difficult to print with. Surprisingly, it is slightly more flexible than PLA, even though I recall PolyMax PC to be a bit stiffer than PLA. I guess the "Blend" is doing a lot of work.

I would turn the little circular pads of sandpaper into long thin rectangles. Only the centre part of the circle ever touches the bar, because the surface is flat and the bar is curved, so making them "wider" serves no purpose, but making them "longer" means they contact more of the bar.

I would switch to probably a 1:1 gear ratio. I don't really know why I didn't do that the first time.

The size of the large hole is limited by the size of large-bore thin-walled bearing that is easy to acquire. I've used "6708" bearings, which have a 40mm bore. It is rare for me to work with round bar any larger than this, so I'm not worried for now. If I wanted to go any larger I would probably ditch the large-bore bearing and resort to 3 smaller bearings placed around the outside, roughly like the base on my telescope mount:

I wonder if there would be some way to have the springs push the sandpaper fingers from an angle instead of radially. That might reduce the required diameter of the large gear:

You'd probably need to have a "half-cylinder" of sandpaper instead of a flat rectangle, because the angle will change with the radius. Maybe it's not an improvement.

Inspiration

The tool is 50% the "drill-powered through-wrench" which is basically a way to use a drill to turn a nut on a threaded rod that is longer than a deep socket:

And 50% those inside-bore honing tools:

I'm currently "interacting" with the computer using an FTDI cable (basically a USB serial port). I'm still planning to eventually acquire some sort of hardware terminal, either a ready-made product or something I'll put together based around Marco Maccaferri's RC2014 VGA Serial Terminal Kit, which I use and like on my RC2014.

SCAMP boot process

The SCAMP boot process has 3 stages. There's the boot ROM, the kernel, and init. The boot ROM loads the kernel off the disk, the kernel stays in memory to provide system call implementations and loads init, and init (currently) just prints the message of the day and executes the shell, at which point the user is in control.

As measured in the emulator, it takes about 36 million clock cycles to go from zero to the shell, but this can be improved a lot. 60% of the time is spent reading/writing the disk. These functions are prime candidates for optimisation in assembly language. Also, init shells out to cat to print the message of the day, which is really an unacceptable performance cost at slower clock speeds. It would be better to have init read /etc/motd itself.

Boot ROM

The boot ROM is stored on 2 EEPROMs (all SCAMP memory is based on 16-bit words, so there's one EEPROM for the least-significant byte, and one for the most-significant byte) which are permanently mapped at addresses 0 to 255.

The memory card decides whether any given memory access should go to ROM or RAM based on checking the 8 most-significant bits: if any are 1 then it's RAM, otherwise it's ROM. This is accomplished with a pair of 74LS32 OR chips, and a 74LS00 NAND chip:

The first three 16-bit words from the disk are:

1. magic number to verify that the disk contains a SCAMP kernel (0x5343)
2. start address of kernel in memory
3. length of kernel

The boot ROM loads the first 3 values, checks that they look sensible, and then continues reading the requisite number of words until the entire kernel is loaded into memory at the requisite address, and finally jumps to that address to start executing the kernel.

Currently every single one of the 256 available words in ROM are used, and the diagnostic messages are still a bit terser than I would prefer, but it fits, which is all that matters.

Kernel

The kernel has very little to do at boot time. It initialises the serial device (which is probably unnecessary since the boot ROM already did that), finds an unused block on the filesystem (for boring reasons), and executes /bin/init.

init

init has even less to do than the kernel:

system(["/bin/cat", "/etc/motd"]);
chdir("/home");
while (1)
    system(["/bin/sh"]);

But it takes a long time because system() in SCAMP/os is basically a practical joke taken too far: it swaps the running process out to disk, loads the child process, runs it, and then swaps the parent process back in and resumes where it left off.

CompactFlash interface

I chose to use CompactFlash instead of an SD card because the parallel access is much more convenient and performant than SPI. I'm using the CompactFlash card in "True IDE" mode, which also leaves the door open for potentially replacing it with a real hard disk at some point with minimal software changes. Happily, True IDE mode supports 16-bit transfers natively, so I don't even have to worry about combining 2 bytes into a word or splitting a word into 2 bytes. Unhappily, it seems to be little-endian, so the filesystem is kind of interlaced when viewed bytewise. It's tempting to swap the upper and lower byte by plugging them into the bus "backwards", but I think this might just create more confusion than it solves.

I bought a CompactFlash breakout board on eBay to get this working on a breadboard, and I may even use the breakout board on the final PCB because it seems a lot more convenient than soldering the CompactFlash connector myself.

I learnt how to speak to the card by reading a couple of PDFs that I found online:

• CompactFlash Memory Card Interface to the TMS320VC54x from Texas Instruments
• SanDisk CompactFlash Memory Card OEM Product Manual from SanDisk

But the set of information that you need to know is much smaller than either of those PDFs, so I intend to write some more beginner-focused documentation.

Graham told me that if your computer has a long bus (like mine does, and like the RC2014 does), then you may find that newer CompactFlash cards don't quite work correctly. I found that mine would sometimes drop certain values. Usually the same ones, but not in any pattern that was obvious to me. There are some details of the problem in PickledDog's rc-cfcard README:

the fast level transitions of modern CompactFlash are only intended for the short traces between card and interface in a system, and do not play nice with the long paths of the unterminated RC2014 bus

I'm not a high enough wizard to know what this actually means, or why it causes the behaviour I observed. But I took a punt on a much older CompactFlash card (you can identify the older ones by their lower storage capacity), and that one worked straight away, so there might be something in it.

Connecting the CompactFlash card to the SCAMP bus is mostly just a case of connecting up the wires in the right order, but a small amount of glue logic is required:

• IORD- = !DO
• IOWR- = !WR
• CS0- = A3 nand A8

DO goes high when the CPU wants a peripheral to output something, but the CompactFlash card wants an active-low signal, so we just invert it.

WR is generated by the UART card, based on DI (CPU wants a peripheral to input something) and the clock signal. We take this signal from the UART card and invert it for the CompactFlash card.

CS0- needs to go low when we want to select the CompactFlash card. I have dedicated the A8 bit of the address register to selecting CompactFlash, and conditional on A8, A3 selects the first card. (I'm not sure if there will ever be more than 1, but it doesn't hurt anything to provide for more now).

I was initially implementing this logic on the Arduino Uno that also generates the clock signal, and at this point the system booted up, which I liked. It took about half an hour to completely boot though, because the clock signal was only about 20 kHz.

Obviously I need to move the glue logic out of the Arduino and onto physical chips in order to make the real CompactFlash interface. All 3 of those operations can be provided by a single 74xx00 NAND chip, so that's what I tried.

Unfortunately it didn't work. I think the problem is that CompactFlash is based on CMOS logic, which is quite strict about having a good 5v for a logic 1, whereas TTL is allowed to output anything above 2.7v for a logic 1. I tried a 74LS00 and 74HCT00 but it didn't work with either of them. I have ordered a 74HC00, but I think this might have the same problem from the opposite direction: it will generate nice clean CMOS outputs, but will also require nice clean CMOS inputs. So if that doesn't work then I may try to generate these signals using bare transistors, and if even that doesn't work then I'll need to go away and learn harder.

Next steps

After I've got the CompactFlash card working with the glue logic in hardware instead of Arduino code, I will need to debug why input wasn't working when I got to the shell. I think it is supposed to work, and it works in the emulator which very-slightly emulates the physical UART, so I'm not sure what I missed. Hopefully it shouldn't be difficult.

Once the CompactFlash card and keyboard input are working, we're pretty much home free, because that's where the dark magic ends. The rest is just building the wooden case that I keep getting distracted from, connecting up the real power supply, making a real clock source, and fleshing out the system software and libraries in preparation for Advent of Code.

The tutorial walks you through implementing the kilo editor, so-named because it's less than 1000 lines of C. Ported to SLANG (and with syntax highlighting not supported), it currently totals 811 lines. I skipped the syntax highlighting partly because of performance considerations (SCAMP is a pretty slow CPU), but mostly because I felt that colours would ruin the aesthetic.

Here's a demo of the editor:

And here's the source of my version: https://github.com/jes/scamp-cpu/blob/master/sys/kilo.sl.

Using kilo is basically like using nano but with a lot fewer features.

Implementation details

Kilo stores each line as a C string on the heap (realloc()'d as needed), with another dynamically-resizing array storing a pointer to each line. This is good because it is easy, but it does mean there is sometimes a large block copy when a string/array needs to be moved. It seems fine enough in practice though, even on (emulated) SCAMP.

The "real" kilo actually stores 2 copies of each line: one is the contents of the line as they appear in the file, and the other is as they appear on the screen. The principle difference is that tab characters in the file are rendered as a series of spaces on the screen. You might be tempted to skip this, but it results in weird behaviour when the screen scrolls horizontally. Perhaps you could work around that by snapping the horizontal scroll to the tab stops? Maybe that is worth trying.

To save on memory usage, SCAMP kilo only stores 1 copy of each line and generates the display version only when needed. You might think this is a premature optimisation, but consider that the source for the compiler is 27K characters. Even with no extra overhead in kilo, storing 2 copies of that would be 54K, and with the kernel taking up 14K, we've already ran out of address space. (As it happens, kilo currently can't open the compiler source anyway, because other inefficiencies push the total over 64K - I eventually want to fix this).

The real kilo redraws the screen after every keypress, which is totally fine on modern computers but is definitely not going to fly on SCAMP: e.g. it would take more than 2 seconds to do a full screen redraw at 9600 baud. For the SCAMP version I keep track of which lines have changed since they were last drawn, and only redraw those that have changed.

Compared to ZDE 1.6

On the RC2014 I use the ZDE 1.6 editor. It is more feature-complete than kilo, but quite weird, and not well-suited to modern keyboards. The backspace key does what the delete key should do, and to get a backspace you need to type Ctrl-H. Sometimes it writes ^I at the end of a line for no apparent reason (I don't think it's anything to do with tab characters). The key shortcuts are confusing and hard to learn, because they don't match any other software I've ever used.

While kilo is clearly inferior to ZDE in terms of features, it is much easier to use, mainly because the lack of available operations means typing the wrong keys is unlikely to make it run off in a direction you didn't want.

To my untrained fingers, ZDE sometimes feels a bit like trying to use vi with caps lock stuck on. Or like trying to drive a car but the steering wheel operates the throttle, the gear lever operates the brakes, and you steer by winding the windows up and down. It's mostly fine as long as you go slowly and carefully, but if you slip into your usual flow it all comes crashing down.

OK, that's a bit unfair. ZDE is definitely the easiest editor I've found for CP/M. Maybe I should port kilo to CP/M.

Hardware

Apart from working on the editor, I've made a bit more progress on the hardware. I have now got a UART wired up on a breadboard, and communication is working in both directions:

From left to right, we have a 1.8432 MHz quartz crystal and a resistor and a couple of capacitors, forming the oscillator from which the baud rate clock is derived. Then there's the UART itself, this is a 16450, which is equivalent to an 8250 but with support for higher baud rates. Then the little black button is the reset button, and finally there's an Arduino Nano.

The Arduino is there to:

• generate a square wave for the clock signal
• invert the reset signal for the UART (the UART has active-high reset whereas the SCAMP has active-low)
• create the "WR" pulse to tell the UART we're writing to it

The fastest I can bit-bang a square wave from C on the Arduino is about 80 kHz, and SCAMP still seems to work correctly at this speed, which is a good sign. I'll be disappointed if I can't get another factor of 10 on that, and delighted if I can get another two factors of 10.

I'm still umming and ahhing about how I'm going to generate the "WR" pulse without the Arduino.

"WR" is an input to the UART that tells it we want to write to it. The problem is that the UART does not have a clock input, and it seems to take input from the bus at the falling edge of "WR". The closest signal to "WR" that SCAMP generates is "DI" which says we want an IO device to take input at the rising clock edge. Simply connecting "DI" to "WR" doesn't work because "DI" doesn't have a falling edge until the next clock cycle starts. At the start of the next cycle, "DI" and the data on the bus both disappear at the same time, which leads to the UART sometimes (most of the time) not reading the correct input. We need a way to make the "WR" signal have a falling edge when the clock edge rises, rather than when "DI" falls.

I think this shows what I mean:

(drawn with WaveDrom)

I want to generate a signal that looks like "WR (wanted)". A potential solution would be to create "DI & !CLK", but it has the risk of a small glitch at the next falling clock edge while "DI" is still high. Past that, I'm not sure. Maybe I have to come up with something based around some flip-flops? I think it would be fine to pulse "WR" when given a rising edge of the clock, rather than holding it high while the clock is low. Maybe that would be easier. Something like the 74121-based single-stepping circuit in my RC2014 front panel.

I have found working on the electronics quite difficult. On several occasions I have reached a working configuration, made a small change, determined that it did not work, reverted the change, and found that the formerly-working configuration no longer worked, with no way to know whether I have permanently broken something, failed to properly restore the previous configuration, or the previous configuration only worked by accident for some unrelated reason. Electronics really needs better development tooling.

I have splashed out on a DSO (I went for a "Hanmatek DOS1102", which is cheap-ish but well-reviewed and seems fine). Unfortunately I haven't found it to be a great deal of use. For example, if I connect it to the clock signal, the CPU stops working and I have no idea why. The clock signal still looks fine to me:

But evidently the magic pixies know they're being observed and they don't like it. It seems like you can't ever use a DSO to debug anything because the mere presence of the DSO changes the behaviour of the system you're connecting it to. In hardware, every bug is a heisenbug.

The card cage helps to hold the cards in place, and to avoid bending the pins when unplugging them. I also anticipate that it will simplify the construction of the case, because the case won't need to hold the rails. Unplugging the ALU is still pretty hard, because it is plugged into 120 pins.

I still haven't made a clock, so for now the CPU is being clocked by a short Arduino program. I've gone up as high as 10 Hz without observing any instability. I don't yet have any way to infer the state of the machine without looking at the LEDs, so if it went any faster than that I wouldn't even know whether it was working or not. Next steps are to make an adjustable 555 timer-based clock, and interface with the 8250 UART.

The test program just outputs the numbers 0 to 24 to the device at address 0. Each number is generated differently, and tests different things, starting off by just sticking the number in the x register and outputting that, followed by addition, subtraction, bitwise shift, bitwise logic, stack manipulation, unconditional jump, conditional jump, looping, relative jump, and function calls.

There is no actual device at address 0, so correct operation is verified by watching the lights. To output a value to device 0:

• the address register contains 0
• the bus contains the value
• the DI signal comes on, to tell the device to take input

Here's a video of SCAMP running the test program. If you pause the video every time DI comes on, you might be able to convince yourself that it is outputting the correct values:

(The button I have my finger on at the start is the "reset" button).

And here's a look at the wiring of the LEDs on the instruction card:

Bugs

Although I had already checked that everything would work at least 3 different ways (Verilog simulation, running on an FPGA, and the SCAMP emulator in C), there were still a lot of fun bugs to find and fix after I put the PCBs together.

Inverted "enable" pins on decoders

Two 74LS138 "3-to-8" decoders are used, each one decodes 3 bits of the microinstruction word into a selection of 8 possible input or output components for the bus.

(This design does mean it's not possible to output to 2 components simultaneously, but that's not something I want to do very often, and saving space in the microinstruction word was more important to me).

The 74xx138 has 3 "enable" pins. For it to apply an output, all 3 must be enabled. I had failed to notice that 2 of these enable inputs are active-low instead of active-high:

The "E1" and "E2" (pin 4 and 5) inputs have little circles to indicate that the signal is inverted, but I had connected these pins to VCC instead of GND, which means the decoders were never enabled. D'oh! My solution was to pop pins 4 and 5 out of the socket and solder a little wire to GND, and this worked.

bus_in conditional on not using the ALU

The "bus_out" decoding is slightly more complicated than "bus_in". One of the components that can output to the bus is the ALU, but the ALU also needs 6 more control signals to select its operation, which are all inapplicable when the ALU is not in use. For this reason, the bit that selects the ALU (EO) is separate from the "bus_out" bits, and the "bus_out" bits overlap with the ALU function selection:

(See under "Decoding" in doc/UCODE.md).

This means that for the "bus_out" decoder, one of the enable inputs needs to come from EO. So far so good. But when I designed the schematic, I copy and pasted the "bus_out" decoder to create the "bus_in" decoder, and forgot to remove the EO input from pin 6:

This means the output of the ALU can never be consumed, because when the ALU is outputting to the bus, nothing is inputting.

My solution was to pop pin 6 out of the socket, and connect it to VCC with a little wire (also visible in the above pic), and this worked.

IOLH inverted

IOL and IOH ask for the lower 8 bits of the instruction word to be placed on the bus, with the upper 8 bits set to either 0 (for IOL) or 1 (for IOH). All of these signals are active-low, because the enable pins on the buffers are active-low.

For some reason, I decided that I could create IOLH by NAND of IOL and IOH (because, by De Morgan's Law, NAND is just OR but for active-low inputs), but obviously this gives an active-high output. I had a couple of spare NAND gates on the same chip, so my solution was to pop the IOLH output pin out of the socket, feed it into the 2 inputs to another NAND gate (grey wire) to use it as an inverter, and feed the output of that NAND gate into the IOLH pin in the socket (blue wire), and this worked.

I must have been tired or distracted when I designed the instruction card, as it has been the source of almost all of the mistakes!

Sequencer reset glitch

So far, all of the mistakes have been simple logic errors, but the sequencer reset glitch was quite subtle, and I am pleased to have figured out what the problem was.

The first 24 or so instructions in the test program were executing correctly, but the 25th was only executing 3 of its 4 microcode steps. The problem was that the 3rd step used the ALU, and set the NY (negate Y) bit.

The sequencer counts up on every clock cycle. The upper 8 bits of the "micro-program counter" come from the instruction opcode, and the lower 3 bits come from the sequencer. Not all instructions take 8 cycles, so there is a signal RT (for "reset T-state") which is used to reset the sequencer to 0. This signal acts immediately, without waiting for a clock edge, so as soon as RT goes high, the sequencer resets to 0 which resets the "micro-program counter" to 0 for that cycle.

Since the combination of RT with any other control signals is meaningless (RT happens immediately, which precludes anything else), I thought it was quite wasteful to dedicate an entire bit of the microinstruction word to it. I encoded it in bit 11, which is shared with an ALU control signal: if the ALU is in use, bit 11 sets NY for the ALU. If the ALU is not in use, bit 11 sets RT.

The problem is that in real-life, signals don't all change simultaneously. If bit 11 ("NY or RT") comes high even fractionally before bit 15 ("I don't want to use the ALU") goes low, then we very briefly have a state where RT is set. This immediately resets the sequencer to 0, stopping the current instruction and moving to the fetch cycle of the next instruction.

That was pretty hard to figure out, but the fix was easy. I have some unused bits in the microinstruction word (I was thinking they might be useful for controlling something like a hardware stack pointer in the future), so I moved RT to one of the unused bits. Then I just needed to pop the sequencer's reset pin out of its socket and connect it to the new signal from the microcode.

I don't know why this problem didn't arise when running the CPU on the FPGA. Maybe I got lucky? Maybe there is some secret internal clocking going on in the FPGA that prevents the problem?

I think if I hadn't managed to work out what was going on here, I would have just disconnected the RT signal. The computer would work, just slower: every instruction would always take 8 clock cycles, even when it could be done in 3.

Next steps

I think the next thing I want to do is get the UART working on the breadboard. I think it should be straightforward, other than the fact that debugging anything without text output is inconvenient. At least getting the UART working will make everything else easier to debug.

Once I have the UART working I'll be able to start running the clock faster, and then I can think about building the "real" clock instead of using an Arduino. I think I'll want 4 clock sources selectable: manual clock with a switch, adjustable 555 clock with a potentiometer, fixed crystal oscillator, and an external clock input. The clock card would also have the reset button and probably a power switch.

And then finally I'll want to implement some sort of storage. The emulator currently provides a made-up block device that provides 1 byte per input instruction, and I know that this is borderline on being too slow to be acceptable. This leads me to rule out the idea of bit-banging SPI to an SD card, which leaves CompactFlash as the most likely storage medium. I'm not sure how much trouble I'll have interfacing with a CompactFlash card, I haven't found documentation to be particularly easy to find or understand. There are really only 2 primitives I need, though: read a 512-byte block and write a 512-byte block. Once I can handle those 2 primitives, we're done. I should then be able to stick my operating system on a CompactFlash card and boot it up for real!

Software

I've also made some progress on the software. Arguably more than on the hardware, but software is easier for me, so it doesn't feel like as much.

The system is usable but relatively slow, even with the emulator running about 20x faster than I expect the real hardware will. Mostly this is down to a combination of the slow block device interface, slow stack and bit-shift operations, bad code generation by my compiler, and code written by me without appropriate regard for how slow the CPU is.

But it works, which is good. A lot of the simpler Unix utilities are there (in a primitive form, at least): cat, echo, grep, head, ls, mkdir, more, mv, pwd, rm, stat, wc. There's an interactive shell with support for IO redirection and globs. It doesn't yet support pipelines because there's no multitasking, but I plan to hack that eventually by writing the output of one program to a file and providing the same file as input to the next.

There is also a compiler and assembler that runs inside the OS, although no peephole optimiser yet. There are a lot of bugs, but also a lot of it works. Here's an example session, showing a bit of command-line usage, and writing and running a program:

(Spot the deliberate mistake in the glob expansion!)

That session was recorded with the emulator running at maximum speed (so, about 20 MHz), and additionally with idle time of more than 5 seconds removed, mainly the compiler.

Compiling software is painfully slow. It takes over 1 minute to compile any program that includes stdio.sl. On real hardware that will take over 20 minutes, which is too slow to be tolerable, so I'll need to do something about it. I have several ideas for improvements but haven't yet done enough profiling to work out what's best, and getting new stuff working is currently more of a priority than getting it fast.

Writing a text editor is probably the next important step on the software side, so that I don't have to write code correctly, first-time, top-to-bottom, with cat. Obviously I don't really need an editor, because I always get the code right the first time and can easily write it in a straight line top-to-bottom...

The Build Your Own Text Editor tutorial walks you through the implementation of Salvatore Sanfilippo's kilo editor (so named because it is less than 1000 lines of C). Another potential source of inspiration is David Given's qe editor for CP/M, which uses an interesting gap buffer data structure to store the text.

I like the watch. It does exactly what is claimed: it is easy to program using the Espruino IDE. The battery lasts a surprisingly long time (seems like at least 5 days between charges). The display is surprisingly high-resolution: 240x240 pixels doesn't sound like a lot, but it's more than enough for an LCD that is only 24mm wide. It tells the time. Great success, just what I was after.

It costs about £70 which, depending on your perspective, is either a lot of money or not a lot of money to pay for a watch.

Comparison to Moto 360

Last year I bought a secondhand Moto 360 to run AsteroidOS on. I ended up regretting that purchase and selling the watch. The Moto 360 runs a full-blown Linux system (!), and has substantially more memory and a more powerful CPU than Bangle.js. It has a (almost) circular display with full touch support. On paper, it's better than Bangle.js in almost every way.

In practice, the battery only lasted about a day, and the development workflow was just so annoying and time-consuming that I couldn't be bothered with anything more than "Hello world". To connect the Moto 360 to the PC, you need to prise off the (glued) back cover and solder USB wires on to the PCB, then plug it into the PC, work out how to get networking working over a USB modem, develop your program using Qt and QML, then install it to the watch over SSH. Then undo your soldering and stick the back cover back on. Such a hassle. And that's after you've got past the step of replacing the stock OS with AsteroidOS!

AsteroidOS is a great project, but developing for it is more of a time investment than I'm after.

(To be clear: this picture shows how you connect to the Moto 360; the watch pictured here is not the Bangle.js)

Bangle.js is the antithesis of the Moto 360 experience. It's simple. Easy to use and easy to program, and you don't have to feel like you're doing something "naughty" just by trying to run your own code on your own device. I've already fixed a bug in an app I downloaded, and I haven't had to flash the firmware, break-and-enter past a glued-on cover, or even get my soldering sponge wet. Bangle.js for the win.

Hardware

The watch is quite big, but not intolerably so. Here's a picture of it next to the watch I've been wearing for the past 7 or so years:

(The pictured watchface is Morphing Clock Plus)

It felt very big at first, but I got used to it quite quickly.

I'm not quite sure what the "N,E,S,W" markings on the bezel are for. They can't be rotated, and seem to bear no relation to anything else.

The screen can sense touch, but only in 2 zones. A touch on the left half of the screen is communicated to software as a press of "BTN4", and the right half is "BTN5". This is OK, and is sufficient to detect swiping left and right, but it would obviously be better if it could sense a grid of 16 or so zones instead of just left vs right.

The weirdest part of the watch is that the screen is actually square, despite the circular housing. I think this is an improvement over making a watch with a square housing, but obviously it would be better if the display were circular as well. I guess circular displays were too expensive or inconvenient to procure.

(The pictured watchface is Animated Clock)

The watch has a really nice and convenient magnetic charging connector, but it seems like a slightly flawed design. The charging cable has a sticker on it warning you to plug the connector in the right way around! The connector itself is totally symmetrical, so it is just as easy to plug it in backwards as forwards. It does not specify what happens if you plug it in backwards, and I'm too afraid to try. Hopefully it just doesn't charge, rather than damaging anything.

(In case you lose your sticker and need to find out which way around it goes: try to remember "the wire goes the same side as the buttons")

The instructions that came with the watch also cautioned against leaving the USB cable plugged in while the watch is not charging, because the magnets and the exposed pins mean the connector is almost perfectly designed to seek out nearby conductive objects and short itself out on them! There is a laser-cut charging dock available on the Espruino shop. This seems like a good candidate for 3d printing, and indeed there are various designs available on Thingiverse.

I found that the watch is very slow to get a GPS fix. After 3 minutes of waiting, Beer Compass still couldn't tell me the direction to the nearest pub:

I expect the GPS will work better outdoors. Although, it's not like the pubs are open at the moment anyway.

I also found that the BLE range is surprisingly short. I have a Bluetooth dongle which I use for wireless headphones, and I can walk most of the way around the house and still hear my music playing. But for some reason, I can't program the Bangle.js unless I hold the watch within about 30cm of the dongle. The BLE range on the watch is not sufficient to reach from my keyboard to the PC under my desk. I don't know why.

Firmware

The LCD configuration is quite comprehensive. It gives very fine-grained control over the circumstances in which the screen will wake up, including automatically detecting when you rotate your wrist towards your face. I'm not sure what units the "Twist Threshold" and "Twist Max Y" settings are in, or even exactly what they mean, or which direction is positive, which makes these particular settings less useful than they could be.

It also seems as though the "Timeout" settings, despite having no labelled units, are in different units. "LCD Timeout" is set to 10, and the LCD switches off after 10 seconds. But "Twist Timeout" is set to 1000, and it seems more likely that this is looking for twists that are shorter than 1 second rather than 16 minutes.

Apps

There are quite a few apps for the watch available on the Bangle.js App Loader.

Unfortunately the apps do not include screenshots. I guess this is because it's inconvenient to take screenshots of the watch, and the Bangle.js ethos seems to be all about lowering the barrier to entry, so requiring screenshots would be too much work. It's a shame, though, because it makes it hard to know whether you want to use something without installing it first. Fortunately, the barrier to entry to installing apps is also pretty low, so it's not too hard to just test out everything that looks interesting and keep the ones you want.

One effect of the low barrier to entry is that some of the apps are not as polished as you might be used to. This is related to something I wrote about before in the context of 3d printing:

If you spend some time browsing Thingiverse then you might notice that some of the parts are designed very strangely. Parts that have sharp and square edges where rounded edges would look neater, thinner or thicker than appropriate, strength in the wrong places, inappropriate fasteners, and so on.

Your first reaction might be "this is clearly bad, these people are no good at designing things". I think the opposite way of looking at it is more useful: people with no training or experience are now able to design and produce parts that successfully solve their problems. It's not that designing for 3d printing results in worse parts, it's that the barrier to entry for design of working parts is much lower than it ever has been in the past. And I think that is a good thing.

Exactly the same effect is apparent in Bangle.js, and I still think it's a good thing.

One example of an under-polished app is the Counter app. It helps you count things. A number is displayed in the centre of the screen, and you can increase it or decrease it using the side buttons or the touchscreen.

It worked, and was perfectly adequate for counting things. But I noticed that when the "widgets" at the top of the screen are hidden, the counter disappears. Eh? Also, the counter is sometimes displayed in the colour of one of the widgets instead of white.

Development

Fortunately, the low barrier to entry of the Bangle.js ecosystem allowed me to quickly and easily fix these problems myself. (The bug was that the app only set the drawing colour once at startup, instead of every time it draws the screen - if the widgets are redrawn and leave black as the active colour, then the counter information is drawn in black as well, and similarly if the widgets leave another colour active then the counter is drawn in that colour).

I was disappointed to find that the Espruino web IDE can't connect to the watch from Firefox. You need to use Chromium. The reason is that Firefox does not implement "Web Bluetooth". I think I'd prefer a local development envrionment to a web IDE anyway. There is an espruino CLI tool, but it's "very javascript", which means it has a million dependencies, and half of them consider various parts of my system deprecated and therefore don't work properly.

It is possible to run the web IDE locally in electron, which I'll probably look into.

I know that I want to write some apps to run on the watch, but I haven't yet figured out what I want them to do. I think a chess clock watch face might be doable, if not entirely practical. I also think a seasonal clock face might be fun.

I'm not completely sure about the state of the BLE security. It seems as if the default condition of the watch is 100% open, and anybody within (the admittedly short!) bluetooth range can run arbitrary code on the watch with no user interaction required. I think it might be useful to write an app for the watch that keeps BLE switched off except when explicitly connected, and times out after 30 minutes or so. You'd then perform some incantation to tell it you want BLE on, and it would allow BLE for 30 minutes before switching it off again. There is a setting in the menu to turn off BLE, but if you forget to turn it off, you're obliviously walking around with arbitrary remote code execution on your wrist. There is an option in the menu to whitelist specific devices:

But I don't know enough about BLE to know if this is foolproof (e.g. can you spoof it? how long does it take to brute-force the set of all likely device IDs? I should look into this). Regardless, I'd be more comfortable with BLE simply off while not in use.

Conclusion

Running JavaScript on a microcontroller is a really interesting way to make hackable devices. Compared to native machine code programs, the JavaScript model makes it substantially easier for unrelated apps to coexist happily, and even run at the "same" time (e.g. the widgets). Nothing is going to trample over the top of the memory of another app when it isn't working directly with pointers, and nothing is going to hog the CPU when everything is event-driven (where an Arduino project would use a while loop, or the loop() function, Espruino apps use setInterval()).

The downside of running JavaScript is obviously the reduced performance and increased memory usage, but I think it is more than worth it. It's a perfect example of the worse is better model, applied to hardware design:

Hardware quality does not necessarily increase with functionality: there is a point where less functionality ("worse") is a preferable option ("better") in terms of practicality and usability. Hardware that is limited, but simple to use, may be more appealing to the user and market than the reverse. from Wikipedia, but with "software" changed to "hardware"