Factorio: Remote Oil Edition

I haven’t posted since May 2018, breaking my plan for a post every month. Let’s catch up, I’m now 7 posts behind. So this is the post for June 2018, about when I started playing Factorio.

So let’s talk Factorio. Specifically, let’s talk the math of turning crude oil into material goods. The usual goods I care about are lubricant, solid fuel/rocket fuel, batteries, processing units, raw sulfuric acid (for uranium mining), and plastic.

My usual method is to place pumpjacks on oil fields, then use a train to ship the crude oil to a refinery at some central location (i.e. the starting location), where I then turn it into each of the things I care about. Similarly, all of the accessory materials (coal/iron/copper) also get shipped by train to the central location, where I make all of my things.

This works okay, but in some cases clearly involves an excessive number of trains coming to my main base. To take solid fuel as a quick example: To make a cargo wagon full of solid fuel (2,000 units. From light oil, of course), you need 20,000 light oil, for which you need 44,444 units of crude oil, or about 1.8 fluid wagons full (with advanced oil processing). It gets worse if you want rocket fuel – a wagonful of rocket fuel (400 units) requires 4,000 solid fuel, which requires just over 3.5 wagons of crude oil.

Notice that I’m ignoring the heavy oil and the petroleum gas from the advanced oil processing step, as those products would presumably be used to make other things.

Also notice that I’m counting everything in wagonfuls, not itemfuls. I posit that the fundamental unit of efficiency in a large base is the wagon – a mega base (or distributed base) would presumably receive 1 wagon of rocket fuel rather than 3.5 wagons of crude oil along with having to hold all the equipment to do the conversions (space is unlimited, except when it occupies space where you want to put other things like assemblers).

I’m assuming that it’s simpler to ship oil to a processing yard for processing before shipping the results to the main factory. There is an extra step, in that you’re splitting what would’ve been one rail line (“move crude oil to factory”) into two (“move crude oil to processing” and the more complicated “move end products to factory”), but the flow into the factory is lower and thus simpler.

But, TL;DR is that here we’re optimizing for the somewhat vague target of “fewest wagon targets into your main factory.”

Moving on. For making a wagonful of lubricant, you need a wagonful of heavy oil, so no real gain there. But, a wagonful of heavy oil requires 10 wagonfuls of crude oil! (let’s say water is easy to access wherever your processing plant is, otherwise everything gets really complicated)

Let’s do something a little more complicated: plastic. A wagonful of plastic (4,000 units) requires 40,000 petroleum gas, which in turn requires 72,727 crude oil – or 2.9 wagons. But! It also requires 2,000 coal, or 1 wagonful. So if you move coal from your factory to your processing plant, you’re only saving 0.9 wagon-movements per wagon of plastic. But, if your processing plant is distant from the factory (as we’re assuming), then you can ship the coal directly to the processing plant, changing what would’ve been 2.9 crude oil wagons + 1 coal wagon into your factory into a mere 1 wagon of plastic to your factory, almost a 4x improvement.

Okay, let’s do sulfuric acid next. 1 wagon (2,000 units) of sulfer requires 30,000 units of petroleum gas, or 54,545 units of crude oil (2.18 wagons). A wagonful of sulfuric acid consumes 2,500 sulfer (1.25 wagons) and 500 iron plates (0.125 wagons), or 2.725 wagons of crude oil and 0.125 wagons of iron.

That’s not too much of a savings, unless you’re willing to also ship copper in. A wagon of batteries (8,000) consumes 160,000 sulfuric acid (6.4 wagons), 8,000 iron plates (2 wagons), and 8,000 copper plates (2 wagons). Those 6.4 wagons of sulfuric acid would’ve required 17.44 wagons of crude oil. By producing batteries at an outpost, you’ve reduced over 21 wagons going into a central outpost to a mere 1 wagon, with ~5 wagons going to your processing outpost.

Processing units are a little more complex – a wagonful (4,000) requires nearly 20 wagons of green circuits, so it’s more efficient for you to move the sulfuric acid to the processing unit factory (only 0.8 wagons of sulfuric acid) instead of vice-versa.

A wagonful of explosives requires half a wagon of sulfer and half a wagon of coal. No particular gain in building explosives at your remote outpost vs. shipping in sulfer. Half a wagon of sulfer adds up to only 1.09 wagons of crude, so there’s a minor advantage over shipping in crude oil.

However, if you plan on building explosive cannon shells, a wagonful (8,000 units) consumes 16,000 explosives (8 wagons) and 4 wagons of plastic, along with 4 wagons of steel, so manufacturing a wagon of explosive cannon shells saves 16 wagons of goods coming to your central factory.

The math can go on. And of course, there’s the added complication of arranging your rails to go every which way. But, if you want the elegance of not having to deal with chemicals at your main base, distributed oil may be the way to go.


Posted in Uncategorized | Leave a comment

Introducing: bdmCore

When I moved out, I thought I was pretty much done with hardware projects. I hadn’t done any in the year prior, but I had done a lot of software projects. So when I moved, I left behind a lot of my bulky hardware, like my oscilloscope.

As anyone who follows my blog knows, that prediction turned out wrong.

It turns out that PCB production costs have fallen, and my salary has risen, to the point where I can afford to custom-make PCBs. jlcpcb.com has a deal where you can build ten boards (under 100mm by 100mm) for $2 + shipping, which if you’re willing to take the slow boat (which took a full month for me!) comes out to a grand total of only $10, or a dollar a board.

This opens the door to a variety of possibilities. In order to learn about making PCBs, I invented a small toy project: The driver timer. The driver timer is a button and a LED that you attach to your steering wheel. You push the button, and two seconds later the LED flashes. You use it to time the distance between you and the car in front of you: You should maintain a two second distance. I’ll discuss that in more detail in a future post.

The important thing for this post is, the driver timer is driven by a microcontroller. When I was growing up, before the “maker” movement got mass audiences, I programmed PIC microcontrollers (to this day I have a few PIC16F84s lying around). These were nice, but required Microchip’s proprietary and expensive programmers. I want to avoid proprietary systems as much as possible.

After PICs, when I next encountered microcontrollers they were in the form of assembled green boards with ARMs that plugged into your computer via USB. These don’t require expensive programmers, but it’s difficult to integrate them into applications because of their pre-assembled form.

So who else makes microchips? Turns out NXP (formerly Freescale) does. (I know someone whose Dad works there, so I knew they did something involving microchips) Mouser will sell you an 8-pin SC9RS08KA2 microcontroller with 4 I/O, 63 bytes of RAM, and 2KB of flash for just over $1.

And, best of all, it has an open specification for their in-circuit programmer/debugger. All I had to do was build it.

Introducing: bdmCore.

bdmCore is (almost) everything you need to program and debug a RS08 (or, as far as I know, any S08-derived core that speaks the Background Debug Protocol). It’s written in a platform-agnostic way, so that it can run on any FPGA. The tradeoff is that you then have to adopt it for your favorite FPGA. (as long as you can hit roughly 50MHz timing, you’re set)

I used a Mojo v3, because that’s what I have. If you just want a Mojo bitstream, hit me up and I can send it your way.

The protocol is setup using four wires: power, ground, BKGD, and RESET/Vpp. (BKGD is the “one wire” in the “one wire protocol”) bdmCore assumes control of all four of these wires, as such there are three control lines you need in your programmer schematic: Power control, Vpp control, and BKGD. BKGD needs to have a pullup to power. Vpp should be 12V (check your datasheet).

Here’s the schematic I used:

Note that you only need Vpp for programming the flash. If you only want to debug, you don’t need that line (or a 12V supply) at all.

The bdmCore FPGA firmware speaks a serial protocol, on the Mojo v3 this means that you talk to it through the USB serial connection.

I won’t delve into the details of how all this works at the low level. But let’s instead do a walkthrough of the code for my carflasher project firmware, to give an idea of how to program and debug a device.

import itertools
from rs08asm import rs08asm
from mc9rs08kaX import *

Bah, standard include stuff to get the compiler and the device. We’ll circle back on this. In fact, let’s start much, much lower in the code, at line 91:

device = mc9rs08kaX(2)
p = rs08asm(device)

This configures what device we’re using (the “2” variant of the mc9rs08kaX device) and initialized a program in that device, “p”. The device is what gives us all of the clever string labels we’re going to use later for registers and memory locations, like “.RESET” and “.PTAPE”


RS08 devices, when they start up (“power-on reset”), begin executing the instruction at memory address 0x3FFD, which because we set the device, we can reference using the handy label “.RESET”. If you’re used to assembly, and don’t recognize this, rest assured – these commands do in fact correspond to assembly instructions. We’re just using Python to wrap the assembly (and to do other cool things, as we’ll see later).

Since 0x3FFD is the last address in memory, the first thing we want to do is jump to start. Hence p.jmp(“start”).


Here’s where “start” is defined. It’s wherever “.FLASHBASE” is for our chip.

# Set the trim NOTE! This value will vary per chip
p.movi(98, ".ICSTRM")
p.movi(0x01, ".ICSSC")

# Set the prescaler to divide by 8
p.movi(0xC0, ".ICSC2")

Curious what the possible instructions are? They’re enumerated in rs08asm.py. The first value in the tuple is the python function name, the last is the layout of the bits, and the ones in the middle are the arguments to the function. Here’s the possible argument types:

  • 8i – 8-bit immediate value, can be a label.
  • 4a/5a/8a – A 4/5/8-bit address, can be a label.
  • n – A literal value between 0 and 7, inclusive. Cannot be a label.
  • rel – A relative address.

For semantic meaning, reference your local datasheet. Note that some otherwise ambiguous instructions have a “i” or an “a” suffix, to indicate it is the “immediate” or the “accumulator” version, respectively. So:

p.mov(98, ".ICSTRM")

Moves the data at address 98 to the ICSTRM register, whereas

p.movi(98, ".ICSTRM")

Moves the value 98 to the ICSTRM register.

The above code is setting up the clock. The internal reference clock can be trimmed. But, of course, the question is, how do you get the trim?

Fortunately, with bdmCore, you can use writeByte commands to set the trim and read back the clock speed indirectly by reading the sync time. This is already packaged for you in sw/trimmer.py in that project. As a fun tangent, you can make a graph of the clock speed versus the trim value, and you get something like:

(graph made with MC9RS08KA2 chip with prescaler set to /2)

Interestingly, not linear. Also shows that at very high speeds (low trim values) the clock becomes unstable. As you can see you get fairly decent precision, with a /2 clock frequency ranging from 3.25MHz to 7.25MHz (a maximum of about 6.5MHz to 14.5MHz).

Anyway, so I ran the trimmer on my chip and got 98 as the optimal value, so that’s what I set.

# Kill the watchdog, enable stop mode, enable BKGD and RESET
p.movi(0x23, pageAddr(p, ".SOPT"))

# Disable & acknowledge low voltage detect
p.movi(0x40, pageAddr(p, ".SPMSC1"))

# Setup the I/O direction and pulldown
p.movi(0x20, pageAddr(p, ".PTAPUD"))
p.movi(0x20, pageAddr(p, ".PTAPE"))
p.movi(0x13, ".PTADD")

More routine setup instructions. If you’re still reading this, it’s because you love poring over datasheets, so go pore over a datasheet!

But notice the pageAddr() call. p.movi() maps to the “mov” instruction under the hood, taking an immediate value and an address. Above we used string values to refer to registers, now we’re using a function?

So you can pass a raw number to movi if you want, but (because of the instruction set) that number has to be less than 256. “.SOPT” is in the register bank residing at 0x200, well above 256, so we can’t simply call p.movi(…, “.SOPT”).

The RS08 architecture has a “paging window”, meaning that there is a fixed set of addresses (0xC0 through 0xFF) which, when read, return the value at some offset in memory specified by the “.PAGESEL” register. Let’s go look at the pageAddr() function:

# Note: This changes .PAGESEL!
def pageAddr(p, addr):
	addr = p._labelOrLiteral(addr)
	if addr < 256:
		raise Exception("You don't need to use pageAddr to access %s"%addr)
	p.movi(addr>>6, ".PAGESEL")
	return 0xC0 + (addr&0x3F)

Aha! So this function goes and resolves the addr you pass it to a number using _labelOrLiteral. Then, it sets the “.PAGESEL” register with the high bits of the address you want, effectively setting the page you want to access, and then it returns the address within the paging window that points to where you want to access.

(Interestingly, this means that if addr is a user defined label that the user hasn’t defined yet, this will error. That’s a bug. Hmm…)

So, if you call pageAddr(p, “.SOPT”), the “.PAGESEL” register will be set so that the paging window points to 0x200, and the address returned will be the location of “.SOPT” within this paging window.

That’s just the beginning of the clever Python code generation. Next we call the function mtimWaitSpan:

# Light each of R,G,B for 1/2 second
mtimWaitSpan(p, (1,0,0), 64, 0.5)
mtimWaitSpan(p, (0,1,0), 128, 0.5)
mtimWaitSpan(p, (0,0,1), 255, 0.5)

mtimWaitSpan lights up the given color of LED for the given duty cycle (out of 256) for the given number of seconds.

I’m going to cut to the chase and say that mtimWaitSpan is a thin wrapper around mtimWait:

mtimWaitInstances = set()
def mtimWait(p, color, brightness, timeSeconds):
	if brightness < 1 or brightness > 255:
		raise Exception("Brightness must be a reasonable value")

	numIters = int(timeSeconds * 256 / MTIM_PERIOD)
	if numIters < 1 or numIters > 255:
		raise Exception("'%s' is out of range with '%s' iters"%(timeSeconds, numIters))

	fnname = "mtimWait_%s%s%s_%s_%s"%(color[0], color[1], color[2], brightness, numIters)
	mtimWaitInstances.add((color, brightness, timeSeconds))

What’s happening here? First some error checking, but that isn’t interesting. Then we set the fnname variable to some long string. That string is basically a hash of the function call – so different invocations get different strings, unless they have the same arguments in which case they get the same string.

For example, the call:

mtimWaitSpan(p, (1,0,0), 64, 0.5)

Gives us a fnname of “mtimWait_100_64_somenumber”.

What do we do with the fnname? We JSR (Jump SubRoutine) to it.

But it isn’t defined! Ah, but we add it to the variable mtimWaitInstances, also. Then, in this code at the bottom:

for c,b,t in mtimWaitInstances:
	mtimWaitCode(p, c, b, t)

is where we define all of the fnnames, by calling mtimWaitCode on all of the mtimWait functions we called. Let’s look at mtimWaitCode now:

def mtimWaitCode(p, color, brightness, timeSeconds):
	numIters = int(timeSeconds * 256 / MTIM_PERIOD)
	fnname = "mtimWait_%s%s%s_%s_%s"%(color[0], color[1], color[2], brightness, numIters)

	# Accumulator counts down from numIters

The first two lines are just repeating the logic to compute fnname, because I’m a lazy programmer. Then we have p.label(fnname) – this is the magic that defines the start of the subroutine we jump to.

This is basically a hacky homebrew linker. One could think of mtimWaitCode as generating an entire library of functions, and mtimWait is generating function calls into that library. Then the for loop at the bottom is inserting the library code (generated by mtimWaitCode) at the end of the program.

And that’s the sort of cleverness that writing assemblers with Python as a framing language allows. You’ll notice that the rest of the mtimWaitCode is coded at compile time – no RAM is used to pass arguments, for example. In fact, let’s just run through that code real fast:

	p.movi(0x04, ".MTIMCLK") # Divide the bus clock by 16

The MTIM is the modulo timer. It runs off of the clock with a prescaler. There’s a register you can set so that when the counter hits the value in the register, it sends an interrupt. This code just configures the prescaler, and loads numIters into A (notice the “i” suffix to “lda” – p.lda(numIters) has a different meaning).

Note also that numIters is calculated at compile time, so is resolved to a number when this code is compiled. As far as the assembly is concerned, it’s an immediate value.

This code has a loop, which for numIters iterations will

  1. Turn on the LED
  2. Wait for “brightness” counts on the modulo timer
  3. Turn off the LED
  4. Wait for 256-“brightness” counts on the modulo timer, so that the total time inside the loop is 256 timer counts.

Or, in code:


	# Run the mtimWait inner loop once
	# Turn on all LEDs
	setColor(p, color)

	# Configure the MTIM for the uptime
	p.movi(brightness, ".MTIMMOD")
	p.movi(0x60, ".MTIMSC")


	# Clear the LEDs
	p.movi(0x00, ".PTAD")

	# Configure the MTIM for the downtime (writing to MTIMMOD also clears TOF)
	p.movi(256-brightness, ".MTIMMOD")
	p.movi(0x60, ".MTIMSC")


	# Clear TOF and disable the MTIM
	p.movi(0, ".MTIMMOD")
	p.movi(0x00, ".MTIMSC")

Nothing too fancy. Note that after waiting, an interrupt will continue execution after the wait instruction.


This is where we decrement A, check if it’s zero (again! note the “i” suffix), if it’s not zero loop around again, and if it is zero then ReTurn from Subroutine.

I’ll leave parsing the rest of the assembly as an exercise for the reader.

Let’s take a brief look at the chunk at the bottom, though:

if __name__ == "__main__":

	def doProgram():
		from programmer import Programmer
		programmer = Programmer(device)


After you’ve added all of your instructions to your program p, you call p.assemble(), which returns to you two things: Your memory map, and some stats. The memory map is a dictionary where the keys are addresses and the values are the memory set at that address.

The stats looks something like:

   'flashUsed': 347,
   'flashAvail': 2048,
   'flashRows': {
      0, 1, 2, 3, 4, 5, 31

Basically, telling you how much flash is used and which rows are used.

The doProgram function takes this data and passes it to a Programmer instance, which contains all of the logic necessary to mass erase your device and program it. (for the curious, programmer._slice(p) returns the rows that will be programmed and the data they will be programmed with)

And voila! You have a programmed chip.

Now, once it’s programmed, all you have to do is debug it…


Posted in Uncategorized | Leave a comment

Great Scott! Software Defined Radio!

I come from a long line of radio engineers (my father’s brother, his father before him, and before that they hadn’t invented radios, so they were just normal engineers). I’m even licensed as a hamateur radio operator, KF5NJG. However, I’ve never operated on my own, seeing as I have never had a radio.

This has changed – for my birthday I received a HackRF One software-defined radio.

I could do an unboxing video. But I already unboxed it, and if the coolest thing about something is taking it out of the box, you need cooler things. Or a cat.

So, following Michael Ossmann’s great SDR tutorials, I decided to decode my apartment’s gate-opening code…. (as part of the homework for one of his tutorial videos)

The first step was to check the FCC database. Apparently the FCC stores the frequency that basically every devices operates on, here for my garage remote.

So, it’s licensed at 318MHz. Time to go look at what’s there.

With a little bit of GNU Radio Companion-fu, I managed to dump an amplitude-modulated waveform the gate remote was transmitting:

That’s all the code required to capture the complex (I and Q) samples from the HackRF One and do two things: 1) send it to a FFT plot and 2) determine the amplitude modulation and send it to an oscilliscope-like plot.

With some speedy clicking and some luck, I managed to freeze the scope frame on a single packet that the remote was transmitting (it continuously transmits while you hold the button, making it easier):

You’ll note that in the scope on top you can see the end of the first packet, a second packet, and then the start of a third packet. The packets repeat roughly every 70ms.

The user’s manual for this device (also from the FCC website) advertises a 16-bit fixed code per remote, plus a 4-bit “facility code”. The question being, how are these encoded in the scope signal?

On the back of my remote is printed, fairly prominently, “FC001 SN03286”, which (and here’s the leap of faith) *could* mean “serial number 3286”.

If you want to try to decode the waveform, now’s your chance! I’m about to spoil it!

From the leap of faith, and a little trial-and-error, the scheme I came up with is that the packet is divided into roughly 3ms bittimes. Each bittime starts and ends low (off), and contains a ~1ms pulse either at the start or the end of the bittime, indicating a 0 or a 1, respectively.

If you do that, this is how you can partition the bits:

…and there’s a match for 0000’1100’1101’0110 = 3286, the serial number printed on the back. (and, the first 4 bits are 0001, which equals the “FC001” printed next to the serial number)

I haven’t tried to replay the data to open the gate yet, or sit out by the gate and record everybody’s gate codes, but that’s just a matter of time.



Posted in Uncategorized | Leave a comment

Vroom Vroom, or, The Lie Of Competitive Indoor Electric Go-Kart Racing

So over Thanksgiving weekend my family and I went go-kart racing at K1 Speed, a multinational chain indoor electric go-kart racing establishment.

I too didn’t think that indoor go-kart racing was a thing. I stand corrected.

Before you read any of the things I write, I want to say that I enjoyed the experience. It had an appropriate mix of adrenaline and skill. I walked away bruised and bloodied, but still had lots of fun (just kidding on the bloodied part).

Basically, the premise is you drive around the track in a kart. You try to drive as fast as possible. They have electronic devices that measure how fast you go. Then, they rank you based on how fast your fastest lap was.

After the race, they upload your scores to a portal on the Internet. You get unfortunately named “ProSkill” points, and they maintain a leaderboard to see who is the best person.

We went around the track three times. I beat my brother, Jason, twice, and he beat me once. The time he beat me, though, he only beat me by 0.320 seconds, out of a 31-second lap. That’s approximately 1%.

So we got to thinking – there’s no possible way that the karts themselves are accurate to within 1% tolerances. These things go through incredible amounts of wear, ranging from the electrical system (the battery withstands approximately 5 rapid charge cycles every day, and puts out 15kW) to the smooth rubber tires, which grip the concrete like superglue and shreds rubber like a pencil eraser.

They post the scores for every race on the Internet. So, off to Scrapy to download them all.

Their website is composed of two different types of pages, “customer” pages and “heat” (race) pages. Each customer gets an ID, as well as a page on each location’s domain name (each of their ~36 locations has a different domain name). Each heat gets an ID that is unique to that location. For example, heat 154614 at Austin is a different heat than 154614 at Dallas. However, player 10262567 at Austin is the same player as player 10262567 at Dallas. To download all of the pages, we want to use player pages to download heat pages, and use heat pages to download player pages (they link to each other in that way, but there is no way to iterate through all players, for example). I’ll leave out the details of parsing the data as an exercise to the reader.

Since, as web scrapes generally go, this process would be interrupted constantly, I decided to use the builtin HTTP cache. This was a mistake. It does not scale well, at least not on my filesystem. The way the builtin cache works is it hashes the request, and then saves six files under a folder with that hash as a name. The six files are the request and response bodies, headers, and other metadata. Retrieving any cached request requires opening and reading multiple files, as well as retrieving the filesystem metadata to get last modify time (if you care about cache expiration). Checking for a response, likewise, requires checking to see if the right folder exists.

The problem with this approach, is after ~6 million pages (K1 Speed boasts that millions of players have played there – a very conservative guess of 1 million players + 100,000 heats per location gives us about 40 million pages across all sites). For one, ext4 filesystems allocate 4KB per file even for very small files, such as request heaters and metadata, so a single cached response is guaranteed to be at least 24KB (4KB * 6 files). On top of this, each file uses an inode, so after 6 million pages my filesystem ran out of inodes long before it ran out of space.

So between the performance hit (of constantly touching the filesystem) and the running out of space, I realized I would have to do better. Instead of creating 6 files for each request, one thing we can do is pack all requests into a single giant file. As a reasonable format, one can (for each record) spit the hash, then each of the 6 record components each preceded by an integer denoting the length of the data.

This makes it easy to cache data (just append to the end), but what about retrieving from the cache? When we want to retrieve a cached request, we don’t know where in the file it’s stored. So, we can trivially keep a giant dictionary mapping the hash to the offset in the file. For good measure, we can memory map the file, to make accessing it fast.

Note that I’m memory mapping a file that I’m appending to. Generally the memory map won’t have access to the newly appended parts of the file, but this is fine – I don’t worry about cache expiration, and the de-duplication filter in Scrapy will kick out everything not in the mmap, so I don’t have to access it.

Crisis averted.

Ordinarily, I have a “I tried this, but then it sucked, so I tried this”, but this actually worked pretty well, with minor iterative improvements. Indexing by brute force at the start of scraping takes ~10 minutes for a 50GB dataset, and the dictionary is only GB in size, at most.

Most of the memory usage was in fact storing the request queue, but that can be stored on disk.

That issue solved, the other issue was that Scrapy’s queue is domain-agnostic. I’m pulling from 36 domains – if one of them is slow, the others can still run. However, if the request priorities aren’t right, then the queue can fill up such that the workers are only pulling requests for a single domain, and if the max per-domain concurrency is hit they won’t try to go find a request for a different domain. They’ll just wait.

I spent some time tweaking the priorities. I found out that if you store your request queue on disk and don’t have bounded priorities, then you will get a “too many open files” error eventually.

But, I did eventually get the data. 54GB of raw gzip-compressed HTTP, 6GB (13 million records) of parsed JSON data. 205 thousand players who played a race in Austin. 97 thousand races over approximately ten years, comprising 655 thousand kart-races (two karts in one race is two kart-races, as is one kart in two races).

Peanuts compared to some professions I can name (most, really). But hopefully enough to answer the question.

To the heart of the matter. I wrote a script that computes the average “best lap” time of each kart over all of 2017 (add up the times, divide by the number of races the kart did in 2017). Karts generally did about 2000 races from 1/1/2017 to 10/31/2017, or an average of almost 7 a day. The way karts are assigned appears to be random, so it shouldn’t be that skilled players tend to end up on certain karts, skewing the averages.

There are 50 karts, but karts 41-50 are the kid karts, which are about 30% slower. So I discounted those in all further analysis.

As it turns out, there is in fact a skew of over a second between the fastest kart and the slowest, with a fairly even distribution between 32 and 33 seconds (and a few outliers also). I even made a graph (my favorite thing in the world):

The real question is, of course, does this affect anyone’s rankings?

The answer is, yes, yes it does. I just analyzed the three games my family played, by normalizing each racer’s best lap time against the average time for the kart (so if your kart has a slow average, you’ll increase in ranking, but if it has a fast average, you’ll decrease). In one of the three games my family played, I went from beating my brother by 0.429s to losing by 0.278s. In two of the three games, my sister went from second-to-last place to last place. In two of the games, the person in second place gained almost a half second on the person in first place, in each case going from about 0.9 seconds behind to about 0.3 seconds behind.

So that is the grand lie. Your times are measured to the millisecond, and scores are assigned based on those hard numbers, but the karts themselves are only accurate to the second.


Posted in Uncategorized | Leave a comment

The Last Jedi

Let’s talk about Star Wars: The Last Jedi. By Rian Johnson. Rian Johnson the Great. Rian Johnson who brought his legendary vision to Star Wars and made everyone happy.

Honestly, I have yet to find a negative review of The Last Jedi. So here I will write a negative review of something everyone loves.

There’s already been a major backlash against the movie, by die-hard fans. I am a die-hard fan of Star Wars, but I’m not one of those fans. Vox gave some common reasons that fans didn’t like it, here’s my thoughts on these issues (just to set me apart from Just Another Hater):

  • Too much progressivism: Progressivism is fine, I love it. I come from a college town, we’re all liberals here (even though I lean more libertarian than most of my friends). You want to spend the movie on some women, an Asian person, and an African person driving the storyline? (or Asian-American and African-American, not sure which is more PC, except it’s silly to add “American” in a universe where there’s no such thing) Watch me not care, though it’d be nice if the people who actually got things done weren’t basically all white men.
  • The jokes are too jokey: 100% agree, the movie tried too hard to make me laugh. When Luke said the same line for a second time, we had already seen too many of his antics, some of which were actually funny, and I wanted a real movie with a sensical plot. Also, Vox claims that this is a nitpicky complaint, but let’s be honest – contrived dialog alone would sink any other movie. It shouldn’t be allowed in a movie that netted over a billion dollars.
  • The movie is uninterested in fan theories: So am I. I didn’t watch any trailers. I didn’t give a second thought to Rey’s parents. I was curious where Snoke came from, but honestly, before I saw The Last Jedi I couldn’t have told you a single fan theory about him. So if the movie broke any existing fan theories, I don’t know, let alone care.
  • Individual plot lines/moments don’t make sense: Aren’t plot lines and moments supposed to make sense? Like, isn’t that the goal? Vox says that the most common complaint is the side trip to the Casino. And sure, it was a little out of the way, but it’s like when they go to Bespin in Empire Strikes Back – it was out of the way, but that’s where the plot led. (also Atlantic’s article gave an interesting thematic tie-in about that trip, it’s worth a read) So I don’t mind that plot line as a whole so much.
  • The characters journeys aren’t what was expected: This ties in with the fan theory point above. I didn’t watch any trailers, so I didn’t have any expectations going in. Except, you know, that Princess Leia would die. I didn’t read any of the extended universe novels, I believe canon is the movies (no books, just the movies. I’m still on the fence about the TV shows). Anything else is fair game. Don’t contradict the movies and I won’t care.

Now that you’re convinced that I don’t hate the movie for the classic reasons, let me tell you some actual reasons to hate the movie (also, here’s an article I largely agree with).

  • Superleia: As expected, Leia dies. Kylo Ren shoots a proton torpedo into the bridge of the ship and she gets blown into space. Except, then she comes back to life, and flies superman-style (with the arm out and everything) back to the ship, where she promptly spends the rest of the movie in a coma. I can’t make this stuff up. This is an ability that’s never been seen before in the movies, yet the characters of the movie take it for granted: There’s not even lip service to the idea that anyone was either surprised or amazed. Getting blown into the vacuum of space by a proton torpedo blast 20 feet away is hard on the body, especially for a little old lady. Perhaps an explanation would be nice. Or, since she needs to be dead for out-of-universe reasons and doesn’t do anything later in the movie, maybe just let her go. (though we do still love you, Carrie Fisher)
  • Escape plan: Holdo’s plan as stated was to fly away from the First Order, who knows the alliance is running out of fuel, and then slip out using “cloaked” transports to a nearby planet. I put “cloaked” in quotes, because as it turns out, they are uncloaked in literally a matter of minutes. Sure, someone told them to look, but the plan requires that not a single radar technician or CO on any of the ships, including Snoke and Kylo Ren who are both personally chasing the fleet, look at the looming planet and think “maybe they’re going there.” Here’s the scenario: You’re chasing the rebel fleet. You know they’re running out of fuel, and have only hours left. The hours stretch by. You see a planet in the distance. It gets closer. It sure seems like they’re heading that way. And they only have just enough fuel to get to it. Anybody could draw the conclusion that that’s their destination (Boba Fett figured out the Falcon was going to Bespin, and that was through hyperspace).
  • Opening Battle: I have a history of disliking the Force Awakens battles. Poe, the best fighter pilot in the fleet, gives his squad such valuable advice as “fly at the enemy and don’t be scared.” This battle is no exception. (and what’s with the giant engine on the back of his X-Wing? Is the real reason that an X-Wing can’t destroy a Dreadnought that it isn’t fast enough? Then why not use an A-Wing?) Rewatch the original, they were a professional army (“We’re heading for the target shaft now.” “I’m going to cut across the axis and try to draw their fire.”).
  • Battle of Crait: The rebels are in a corner. There is no escape. The door will hold until reinforcements arrive, until… Oh no! The super mega cannon! It can blast open the door in no time. They need to destroy it, or else the First Order will spend the day deciding who to execute first. They gather the last ditch pilots (apparently including a plumber), and send them out to destroy the cannon. They get picked off one by one from above, but they continue, because they have to destroy the cannon or it’s over. Suddenly, the Falcon appears, and provides air superiority, drawing off the TIE fighters. Now the squad is safe from above, they just have to survive the AT-M6s. One member (Finn) makes a suicide run against the cannon as it starts to charge up.
    And Poe calls off the attack.
    What? But if you go back, you know you’ll die. The cannon will blow the doors open, and you will be tortured by the First Order. Honestly, it’s probably better to die a quick death flying one of those rust buckets.
    But you heard him. Turn back. Now that they aren’t being picked off from the air, it’s too dangerous to press the attack that’s literally seconds away from working. Retreat, and sit in the rebel base for a few minutes until the cannon fires and everyone is wiped out.
    Rose even crashes her own ship into Finn’s, to save his life, dashing their only hope of destroying the cannon. She justifies this by saying “we’ll win not by destroying what we hate, but by saving what we love” before collapsing. How romantic. If I were in Finn’s shoes, I would shoot her there and then, to save her from the First Order welcoming party. Sure, a fine message to send, I’m sure it’s part of Rian’s “vision,” but maybe send it in a less on-the-nose kind of way.
  • Battle of Crait: The First Order commanders are… floating 100 feet above the battlefield, in plain view of turbolasers? They aren’t worried about being shot down? And the rebellion didn’t think to shoot them down? Commanders should be fired on both sides of that.
  • Laser bolts affected by gravity: This one is nitpicky, I’ll admit, but notice that as the fleet is firing on the rebel cruiser the laser bolts are arcing through space. This is not how that works in the Star Wars universe. Sure, I understand that you’re trying to appeal to a bigger tent to make more money, but you are still dealing with an established franchise. Imagine if you made James Bond work for the CIA, because there are more Americans than British people. It wouldn’t be okay. Laser bolts go straight in this universe (the Star Wars universe).

Rian, movie franchises have audiences for a reason. They’re a known quantity. If you watch a Harry Potter movie, you know that you’re getting a teenager fighting evil using magic. If you watch a James Bond movie, you know that you’re going to see a suave British dude kicking ass, getting ass, and generally being a badass. If you watch a Marvel movie, you’ll see a group of superheroes saving the planet from some bad guy who’s apparently even badder than the last guy. If you watch a X-Men movie, you’ll see a group of superheroes saving the planet from some bad guy who’s apparently even badder than the last guy.

When I want to watch one of these things, I know exactly what movie to pick up. Franchises are known quantities, we expect them to be the same from movie to movie.

That’s how I feel about Star Wars. I grew up on the tense space battles. My skin crawls every time I watch the start of the Battle for Yavin, when everyone’s getting in their space boats and flying into battle. The Shepard’s tones of the engines starting up, the ground crews scurrying about, doing their last minute preparations. Call me a nerd, but I love thinking about these logistics. I don’t know what the hundred or so people drawing on the translucent glass screens are doing, but I like to see them work all the same, and I like to imagine what they’re doing.

And why did that have to go? I get that Rian’s trying to make Star Wars appeal to wider audiences, to cash in, but The Last Jedi could still have been good without the military elite making blatantly terrible choices. It could even have kept the same plot, with the same themes. (handoff from older to younger generations, etc.) I’ll be the first to say that I don’t want to rehash “nobody from backwater planet is actually special and saves the galaxy” over and over. But you this is not an issue of high level plot, it’s an issue of writing – The Last Jedi had an amazing plot, a wonderful plot, a plot that I swear by, it was just terribly written. It was written by someone who tried too hard to make a point, at the expense of the movie.

The irony, of course, is that a major theme in the movie is that some of the worst of people aren’t the people on the good side or on the bad side, but the people in the background making buckets of cash while watching the world burn.

The debate over whether The Last Jedi was a good movie has many sides, but at the end of the day the filmmakers (and Disney) are in the background, making a billion dollars.

P.S. Another Rogue One would placate me. Or if you pointed me to a new franchise that I would like. Yes, I’ve already seen the new Battlestar Galactica.


Posted in Uncategorized | Leave a comment

Bechdel Test

So, I was thinking about the “Bechdel test” the other day, since sexism has been the “in” thing for the past few weeks. Also other reasons. Basically, it’s a test to highlight gender inequality in fiction (thank you, Wikipedia, for that description). There are three criteria that must be met to pass:

  1. There must be at least two (named) women in the movie,
  2. These two women must talk to each other,
  3. About something other than a man.

Then you score the movie out of three points. Higher is better.

This is great at first blush, but there are a few problems with it. First off, it doesn’t account for movies for which an active female role doesn’t mean that they must talk to each other. I don’t have any examples, so take this with a grain of salt.

But on the other end of the spectrum, any reasonable pornographic film will have two women that talk to each other (and probably a reasonable number will talk about something besides a man, though I lack the data), and I hardly consider that a particularly gender-equal medium.

Honestly, though, that’s not my point. More on topic for this blog (ha, is joke), there’s a website which provides vast quantities of user-rated bechdel test scores. I downloaded this data and did some simple number-crunching in Python.

The data comes as ratings (on a scale from 0 to 3, of the number of criteria met) for a few thousand movies, each linked to an IMDB ID. Using my TMDB database, I can download things like year and genre.

So a first test, just to make sure everything’s working, is to graph the average movie score against time:

What to make of this? With my many years of sociology research, here’s my interpretation:

  • Early movies didn’t typically have women in leading roles.
  • During World War 1, movies started passing the Bechdel test more and more.
  • Then the Roaring Twenties came, and women went back to their more ornamental state.
  • The Wall Street crash, heralding the Great Depression, and we stopped being rich enough to trample on women in film.
  • Then the ratings held steady until, ironically, second-wave feminism came about in the sixties, where the ratings dropped.
  • From there, the noise in the dataset lowered and the average has roughly steadily increased to present day. A lot of this is just because there are more rated movies from this time period.

That was a nifty graph. I mentioned earlier that I had genre data for all of the movies, what if we find the average Bechdel rating for each genre?

             War 1.449 
         Western 1.467 
          Action 1.762 
           Crime 1.826 
       Adventure 1.856 
         History 1.856 
 Science Fiction 1.903 
     Documentary 1.929 
       Animation 1.981 
        Thriller 2.028 
         Fantasy 2.096 
         Mystery 2.188 
           Drama 2.200 
          Family 2.227 
          Comedy 2.229 
          Horror 2.278 
           10769 2.320 
           Music 2.412 
         Romance 2.431 
        TV Movie 2.587

That’s the list of genres with their corresponding ratings, sorted increasing down. Movies are from all time periods, averaged together.

Right off the bat: Genre “10769” is a genre that, for some reason, TMDB won’t identify in their API (it turns out to be the Foreign genre). It appears to be largely bizarre international films, including the Japanese fantasy drama horror romance action comedy movie “Negative Happy Chain Saw Edge.”

Update from a friend who can read Japanese: That’s not a mistranslation.

But, the rest of the genres are familiar to us Americans. Interestingly, if I were to rank these genres from “most manly” to “least manly,” this is about the order I would put them in (notably, I don’t by default rank them from “most manly” to “most girly”, I don’t think of these as opposite).

On the one hand, you have War, Western, Action, and Crime movies. I thought Saving Private Ryan was a good movie, but to be honest, I’d have been really uncomfortable if half the squad had been female. Saving Private Ryan fails the Bechdel test with 0 out of 3 criteria met. So War movies get a free ticket.

Jennifer Lawrence, you were good in the Hunger Games, but I don’t consider that a war movie in the same vein as Saving Private Ryan. So don’t try to do a remake of that movie. I don’t like imagining my future wives getting hurt.

Tom Hanks, you’re cool, but you’re on your own in this regard. Man up.

Western, Action, and Crime movies are in a similar boat, though much less extreme. Spectre passed only 1 of the 3 criteria, which makes sense, but the movie wouldn’t have been worse had more females talked.

Science Fiction, though, has no excuse. We put a (black!) woman on the bridge of a starship in the sixties, and we still can’t pass the Bechdel Test more often than thriller movies? How can you do worse than the Fast and the Furious movies?

Oh well. I’m not a movie maker. I’m just here to make graphs and complain about society.

Speaking of graphs, what if we look at the difference between the most manly and the least manly genres, on a per-year basis? In principle, in a very unequal world, we would expect there to be a large gap across genres – some genres would be considered manly, and others not. In a perfectly equal society, every genre would have the exact same representation.

So let’s make that graph!

And look at that, in recent years the graph has been trending downwardish for the past 20-30 years. So that’s good, I suppose.

Fair warning: The amount of data per point in this graph is just a few data points. There’s only ~7000 movies in the database, divided into 100 years is ~70 movies/year, divided into 20 genres is only ~3 movies/year/genre.

This issue is compounded by the fact that the movies are not distributed uniformly across time. Early on there were fewer movies than there are today.

I don’t have a graph of number of movies in the dataset against time, that’s left as an exercise to the reader.

Of course, the part that you all are curious about: the code. Boys and girls, this is what real I-have-to-analyze-some-data-real-quick-I’ll-comment-it-later Python code looks like: https://gist.github.com/lkolbly/8d657411cc50e1e01b164778b6239523


Posted in Uncategorized | Leave a comment

The Programmer’s Cookbook: Stuffed Pizza

I don’t really like food. This is surprising, since being a programmer I get exposed to a variety of foods from around the world, ranging from Greece to the Golden Triangle to America (U.S., Canada, and Central America).

I guess “around the world” actually means “the Northern hemisphere” in this context.

And yes, I recognize that what I eat from restaurants near my workplace hardly counts as authentically from those locations. But I’ll take what I can get.

But that said, I like pizza.

In general, my favorite tends to be deep dish pizzas. Super thin (New York style) pizzas are good as well, but in my opinion don’t stack up as well. My long-time favorite is a Chicago style stuffed crust pizza that comes from Mangia’s in North Austin. Recently I was introduced to the Detroit style of pizza (from Via 313, also from Austin in case you had any doubts about where I lived), which prides itself on being made in automobile parts trays, after the car industry they built the city on. These are both delicious deep dish pizzas.

More modern Detroit style pizzas are baked in an abandoned factory with broken windows and rusty parts trays.

Anyway, since I moved and started shopping for myself it’s become easier to cook for myself beyond just warming up frozen pizza. And I’ve found it’s surprisingly easy to make not terrible pizza from some flour and some sauce and some cheese. So I do that, because it’s good food and it’s cheaper than eating out all the time and maybe also cheaper than eating frozen pizzas, or at least about the same cost. Either way it’s tastier (though nobody other than I has taste-tested my cooking, so I could be lying). But, these are simple single-layer pizzas, and I do miss the taste of more complicated pizzas.

So, I took on the challenge of making a stuffed crust pizza. This is my recipe and my results, put in terms a non-chef would understand.

Note: Here one pizza is enough for two people, i.e. suitable for you and a hot date or (more likely) you and future you looking for leftovers. It is assumed neither you nor your hot date are professional pizza-eating-contest-contestants. If so, double the recipe.

Also worth noting is this recipe is currently in early development, I find it to be very doughy. You may want to halve the amount of dough used. I will update as I see fit.

Step 1: The Dough

The result of this step: Two covered bowls full of risen dough. (rised bread? It’s not bread unless you bake it, though)

This part is surprisingly easy, especially if you grew up around bread machines. I had always assumed that bread machines were really important for all things yeast.

And they are. Just not for pizza. You don’t need a bread machine, just a bowl and a willingness to make a mess.

Here’s a list of tools you will need:

  • A mixing bowl, ideally approximately one forearm’s diameter.
  • A 1 cup measuring cup (and a knife with a flat edge).
  • A 1 tablespoon measure.
  • A 1 teaspoon measure.
  • A big spoon for mixing.
  • 2 bowls.
  • Plastic wrap.
  • Thermometer to determine water temperature. (though you don’t have to be very exact)

Here’s the list of ingredients:

  • 3 cups flour. I buy it 5lbs at a time of the store brand, because that’s cheapest.
  • 1 tablespoon of sugar.
  • 1 teaspoon of salt. The one with the little girl with the umbrella is fine.
  • 1 packet of yeast. Most recipes say “1/4 cup of yeast,” but it’s sold in 1/4 cup packets, so just get a packet (actually it’s sold in sets of three 1/4 cup packets).
  • Heatable water. None of that ice-nine stuff.

Here’s the procedure (read before doing!):

  1. Pour 3 cups of flour into your mixing bowl. To do this you can scoop a heaping pile of flour, then use the back of a knife to scrape the heap back into the bag. Measuring cups measure to the brim.
  2. Pour 1 tablespoon of sugar into the mixing bowl.
  3. Pour 1 teaspoon of salt into the mixing bowl.
  4. Take a packet of yeast, and cut the corner off. Empty the contents into the bowl.
  5. Mix the dry goods.
  6. Fill your cup measure with warm (110 degree) water. You can do this using a thermometer for the first few times. I found it’s approximately between “warm” and “hot” water. If the temperature isn’t exact, that’s fine – I’ve never had dough fail to rise because the water was too warm. Remember that a thermometer will take some time to heat up.
  7. As soon as you poured the water in, take your big spoon and start mixing. The goal is to get the dry goods to mix with the water. Generally at the beginning (first few seconds) I will push the dry goods to the edges of the bowl, into the water, with my spoon. Then I will scrape along the sides of the bowl toward me, and then squish the spoon through the middle of the bowl to the far side (in a sort of pressing motion).
  8. After a minute or so, the dough should start to pull together into small strandy dough-like bits. You should be able to reach in with your hands and smoosh all of the dough together. Some bits will fall off, that’s fine. Sometimes if they’re big you can stick them back on. Gather all of the bits you can reasonably hold.
  9. Now knead the dough. Press it like you’re giving someone a massage, except you can physically push your hands through them and they are a ball of dough. I keep it suspended in the air over the bowl. Squeeze it, then fold it back into a ball shape and squeeze it again. The goal is to mix it even better, I guess. Do this until your hands get tired and the floury dough has magically turned into sticky homogenous not-floury dough.
  10. Split the dough in two, place each into a bowl, and cover each bowl with some sort of plastic wrap.
  11. Place each bowl on top of a computer running Tensorflow or other heavy computation (at least 300W with heavy CPU and GPU utilization is ideal). A webserver may be fine as well. (but seriously, the temperature should be at least 78 degrees and ideally 80 degrees for the bread to rise. This can be a problem if you like your apartment cold. 74 degrees is too cold)

Step 2: The Toppings

The result of this step: The toppings I used when I made this post (onions and peppercini).

You can do this step concurrently with the next step if you wish. This is mostly just a placeholder step so that you remember to make toppings, which involves buying them and buying the tools you need.

Here’s a list of tools you will need:

  • Sharp knife.
  • Cutting board. I use one of those small flexible non-stick ones I found at a grocery store.
  • Bowl. Or, somewhere to put the ingredients before you put them on a pizza.

Here’s a list of ingredients:

  • Peppercini peppers (pronounced “pepper-ch-eeny”). Not the similar-looking but different banana peppers. If you have to buy a bunch of pizzas from Papa John’s just for the peppers, do it. You will need approximately 3 peppers for a whole pizza, adjust to your liking.
  • An onion.

Here’s the procedure (Read before doing!):

  1. Cut a thick (1/2″) slice of onion. Remove center, cut onion slice into quarters. Set aside. (note: This actually produces much more onion than is probably ideal, if you’re doing something else that requires half an onion slice then go for it)
  2. Chop peppercinis into ~3/8″ slices. You should get ~4 slices out of each pepper. Set aside.

Step 3: The Pizza

The result of this step: A stuffed crust pizza.

Here’s a list of tools you will need:

  • Cutting board
  • Pizza cutter
  • Rolling pin
  • Pie pan (Mine is 8″ in diameter and 1 1/4″ tall, taller would work but I wouldn’t go shorter)
  • Cookie sheet (note: This is just to roll the dough out onto)
  • Spoon (for spreading sauce)
  • 1 tablespoon measure.
  • Oven mitt.

Here’s a list of ingredients:

  • Dough (both bowls of dough from Step 1 – this is a two-dough recipe)
  • Toppings (from Step 2)
  • 1 tablespoon of olive oil.
  • Mozzarella cheese. So much mozzarella. On the order of 8-12 ounces of mozzarella. I buy the pre-shredded variety that comes in bags at the grocery store.
  • Pizza sauce. This part is being a little finicky with me, you can make your own or you can buy a jar of pizza sauce from the store. It doesn’t provide a majority of the taste.

Here’s the procedure (Read before doing!):

  1. Preheat oven to 450 degrees Fahrenheit.
  2. Roll out 1 pizza dough. It should be about 1/4″ thick.
  3. Roll out the other pizza dough. It should also be about 1/4″ thick.
  4. Spread dough in pie pan. Be careful to press the dough into the corners of the pan. The dough should lie along the bottom of the pan and up to the rim. It may overhang the rim of the pan if you want.
  5. Pour 1 tablespoon of oil onto the dough (not more! Even if you’re tempted). Then spread it around with the back of a spoon (or similar) until it evenly coats the dough.
  6. Place the pizza in a 450 degree oven for 5 minutes to toast the lower crust.
  7. Remove pizza. Place a layer of cheese on the pizza. The layer should be at least 1/2″ thick, consisting of several handfuls of cheese. This layer uses half of the cheese on the pizza.
  8. Place toppings on the cheese layer.
  9. Place the second layer of cheese. This layer should be about the same size as the first layer, and should come to the rim of the pie pan.
  10. Place pizza in a 375 degree oven for 5 minutes to let the cheese melt. Remove pizza.
  11. Add the second crust to the top of the cheese layer. Tear off any pieces of dough that go outside the pie pan. Note that, because of the cheese, this dough may end up well above the rim of the pie pan. Pinch this layer of dough to the edge of the first layer of dough, so that the cheese and toppings have nowhere to escape to. Warning: The pie pan may be hot.
  12. Place pizza in a 375 degree oven for 15 minutes.
  13. Remove pizza onto cutting board. Cut into slices. Consume pizza as desired. Note that the pizza may be messy, so don’t eat it in a bathtub full of cash.


Someday I’ll fix the image rotations, I promise.

So, there you have it, that’s my current recipe. Here’s the fine print: It’s honestly not all that great, both times I’ve made it it turned out to be extremely doughy (as in, the pizza was more loaf of bread with cheese in it than cheese with crust around it). I suspect this is because I use two doughs, I intend to try using one dough and splitting it in two for top and bottom. I will update this recipe as I see fit.

Comment below with your questions and results! Actually, don’t, except as a last resort. If you’re reading this you probably either a) can contact me directly or b) can contact me through somebody who knows me directly.


Posted in Uncategorized | Leave a comment

On Thermostats

So I recently moved, in case you’re new to the blog, which has entailed me becoming comfortable in a new home. Or, making the new home mine.

It has also entailed me learning about apartments. This being my first one, I’m… a little surprised by the cost cutting measures. The apartment’s great, don’t get me wrong, but I think it would’ve been nice if during the recent renovation (which I’m paying premo for) they had bothered to install actual drawer mechanisms that, you know, actually open and close without sticking and going all over the place. I may be slightly spoiled from spending my entire life so far in a house with drawer slides.

But, more to the point, I pay for electricity and my thermostat doesn’t have a schedule feature. It just holds whatever temperature you set. So I could turn it off when I leave in the morning and on when I get back, but then I not only have to spend my first half-hour home from work in sweltering heat but I have to remember to turn it off, which I won’t. I barely remember how to get to my building, how am I supposed to remember to turn off my A/C?

So, solution time. I had a few weeks before work started to think something up. I could buy an off-the-shelf solution, but that’s both expensive and a cop-out and it won’t necessarily work with the installed HVAC system (the thermostat mounts onto a 4-wire header sticking out of the wall, so anything I buy would have to conform to that standard).

My second option would be to rig up some relays to the 4-pin header sticking out of the wall. The problem here is I’d risk blowing out the circuitry that’s hidden behind the wall, and that’s almost certain to not get me my security deposit back. Also I have to deal with all of the weird cases that thermostats deal with – things like compressor protection.

That leaves me with the third option: Build a device to press the buttons on the thermostat for me. And that is what I did, with the help of my trusty 3D printer.

Sorry the image is sideways, I still haven’t figured out image rotation.

As you can see, there are three parts: A servo (partially hidden) to press the buttons, a webcam to view the current temperature, and a Raspberry Pi to control the entire charade. The cost of parts was $0, because I still have all of those Pi 3s from when I built the Pi cluster.

The Pi 3 has WiFi, so I don’t need to run an ethernet cable to it – all I need to do is get 5V USB power to the Pi. This is how the Internet of Things is supposed to look, not to snub the people who use “Nest” or whatever. (machine learning thermostat? What is there to machine learn?)

Building the hardware was, as always, an iterative process, made vastly easier by owning a 3D printer because I can run a print, see if it worked, tweak it, and repeat. Productivity is inversely proportional to the edit/compile/run cycle, and cutting down the weeks and $10s of dollars spent outsourcing builds to hours and pennies has been possibly one of the greatest quality-of-life improvements I’ve experienced of all time.

For example, here’s the five and a half prototypes I printed for the servo rocker arm (the thermostat has two buttons, up and down – the servo is configured so that if it rotates one way it presses up and if it rotates the other it presses down):

The first iteration was the bottom-center one, which was printed such that the support material filled the hole for the servo. Then came the ones on the bottom left and right, which are similar save for a slight change in hole size. Once the rocker fit the servo, I printed the top-right one, which featured a redesign of the arms – the straight arms tended to snag on the edge of the raised buttons, and I didn’t want to damage the buttons so I put a curve on the end of the arms to prevent snag. The top-right one had the curve built such that the ends of the arm stuck together, so when one compressed against a button the other did too, pressing both buttons simultaneously. So I printed the top-center one, which fixed that, but then I realized that the method of mounting to the servo was flawed, since the 3D print couldn’t grip the grooves along the servo head. So, I prototyped a mount for cross-arms, and the final print is what’s mounted. (a nice thing about using OpenSCAD for designing is I can select just a region of the part to print, if I don’t want to waste time and filament printing a whole part just to test one bit)

The rest of the thermostat mount came together in a similar fashion, I won’t bore you with the details. There are two main parts, the left mount and the right mount, which bolt together to secure themselves to the thermostat. The Pi mount on the left half is a shortened form of the Pi mount from my cluster (another nice thing about OpenSCAD – copying/pasting libraries). The webcam mount took some clever design to fit on my printer, it’s exactly big enough to be running against the size limit.

The LED mount was an afterthough. Originally, there was no LED, but night tests showed that the webcam couldn’t read the display well enough, so I added a red LED.

Now that I’ve discussed the hardware at length, let’s dive into the software.

Oddly, the system is open-loop. That is, the system doesn’t make decisions based on the temperature. This is strange because the system knows the temperature, because of the webcam. Instead, the webcam and the servo control are separate entities software-wise.

The servo control works by storing the last-set setpoint in a file, and there is a Python script that checks what the setpoint is and what it should be and moves the servo to match (this is how I get around the open-loop problem: the system just remembers what the setpoint is supposed to be). The actual mechanics of managing the servo (driving PWM values, etc) are done through shell scripts using the wiring-pi command, because for some reason the Python module kept crashing my Pi when I re-initialized the PWM output. So the high level Python script calls out to the shell scripts via the subprocess module to manipulate the thermostat.

Of course, this raises the question of how does the script know what the setpoint should be? First off, a schedule is nice, but how should the schedule be represented, and what should happen for non-scheduled events? In terms of representation, the schedule should probably be able to at least support weekend/weekday differences, because that’s the point of the whole system (I don’t want the A/C to turn off on the weekends, because I’m home then). But should it support having different schedules on, say, Monday than Tuesday? Or Sunday from Saturday? (I don’t go to church, but that’s really the only use case I can think of there) If I do support having different schedules on different days, how do I want to represent that? If I want the same schedule for every day, do I have to copy and paste the schedule? (which would be bad design, because then a change to one day’s schedule won’t propagate to all)

If you like thinking about such things, then congratulations in your career in software engineering, and take a moment to think about how you would do it in your situation.

How I ended up doing it was by building an ordered-override system, where files specify time ranges and apply to specific days (of the week). A file does not need to specify the temperature for all of the times in a day. Files are named like 012-myname, where the number in the beginning is used to sort the files and the name is just a human-readable name. Higher-numbered files override lower-numbered files in areas where they overlap.

For example, here’s a set of configuration files:

pi@rpi06:~ $ cat schedules/000-night  
# This is the default schedule for overnight: 9PM to 6AM 
pi@rpi06:~ $ cat schedules/010-workday  
# This is the monday-friday workday schedule for summer. 
 6:30- 8:00=80 
     -16:30=92 # While we're away for the day 
     -20:00=82 # When we're home 
     -21:00=80 # Gradually decreasing for bedtime 
pi@rpi06:~ $ cat schedules/011-weekend  
# The saturday/sunday summer schedule 
 6:30-11:30=80 # Stay cool for the morning 
     -18:00=84 # Moderately cool for the afternoon 

This is a hypothetical user, not me. Please don’t rob me while I’m away from home.

There are three files here, specifying the “overnight” schedule, the workday schedule, and the weekend schedule, respectively (and, they apply in that order). Note how the overnight schedule applies to all days, but only specifies temperatures from 9:00PM to 6:00AM, whereas the workday and weekday schedules both start at 6:30AM. This means that 6:00AM to 6:30AM is unspecified. In this case, the default is to leave the thermostat as it was (the goal there being to change the thermostat as little as possible).

This schedule system works pretty well – it allows day-level granularity, while also not duplicating schedules unnecessarily (notice how above, the night schedule is only specified once). It also allows temporarily dropping in a new schedule file, if I (for example) go on vacation for a few weeks.

However, it doesn’t account for one thing very well: events and outings. Sure, whenever I go somewhere I could drop a temporary “100-sallys-party” file in, but that requires me to know in advance when I’ll get home (if I want to cool my apartment off before I get back).

Introducing the notion of “holds.” A hold is just a temperature for a one-time range – for example, hold 92 degrees (i.e. turn off the A/C) until 6:00PM today.

This doesn’t immediately solve the problem, except with the addition of two things: The smart home controller and a cellphone.

The smart home controller is just a Pi with a touchscreen mounted next to my door (it isn’t mounted yet, hence no picture). The touchscreen shows some useful facts, like the time and the temperature and the forecast, but it also has a button that says “Leave”, which when pressed submits a 92-degree hold for several thousand hours (effectively forever). This means as I walk out the door I can turn off the A/C with a single press of the button.

That’s handy, but then to solve the opposite problem (cooling down the apartment on my way home) we turn to the wonderful API from Twilio.

Twilio lets you setup a fully featured voice (or SMS) app using nothing but a few bucks and a PHP server. When your phone number receives an incoming call, they send a request to your webserver to figure out what to do, and that goes back and forth (they have excellent documentation you can check out). You can configure it to do the whole “press a button for a menu” deal, so that’s what I did. Now I have a lovely British lady in my speed dial that I can call to cancel a hold (or, receive a weather forecast for the next day).

So that’s it. Press a button when I walk out the door, make a 15 second phone call when I’m heading back. It’s a perfect system.

Well, except for one thing. I haven’t discussed the webcam setup.

Partially because the webcam is much less elegant. Much, much less elegant.

Basically, it can be summed up by: Every minute, the Raspberry Pi captures an image with the webcam (script in a crontab). Every 15 minutes, the Raspberry Pi processes all of the images to read the current temperature and whether the A/C was running.

The image processing script works by comparing the average brightness of fixed regions in the image. For example, I’ve hard coded the position of each of the seven segments within the image. The reason for this is a) it works and b) I don’t have enough test data to come up with a cool machine learning algorithm to do it any other way. The reason for both of these is that the webcam is fixed relative to the thermostat, so it works.

It’s just hackish.

To be completely fair, there is an auto-tuning script which will go through a bunch of labelled samples and figure out the right coefficients for comparing region brightnesses, but I still have to hand-code the locations of the regions.

Either way, once the data are computed, the Pi sends it off to a mongo server running on my main server.

I can query the mongo server to make cool graphs such as this one:

Which graphs the temperature inside my apartment against the time for the past few days.

There are still a few bugs left to work out in the vision system. For one, the data is a little hard to read since it comes back on integer degrees. For two, I recently hung a Star Wars poster above my couch, in the reflection of the webcam, which is causing some spurious readings. Sometimes it’s hard to predict where bugs will come from.

The source code is a bit gnarly, paths are hardcoded, and so on, so I haven’t posted it online, but it is open-source. In the sense that if you ask me for it, I’ll happily post it.


Posted in Uncategorized | Leave a comment

…and moved.

Welcome back to month two of Lane Kolbly’s “I have a blog, I should probably write in it, but I’m tired and doing other things so I’ll wait until the last possible minute.”

In my defense, I moved and there was a hurricane 100 miles away from where I lived. It was quite nice (I like rain, at least when I can curl up under a blanket and watch movies while it pours outside. I’m not so fond of interacting with the rain).

So I thought I’d take this time to briefly revisit a topic that’s dear to my heart, sexism in programming, and share an interesting epiphany I had about my psychology (though I’m neither a shrink nor very self aware, I think that’s the right way to say that).

Just to clarify, I think sexism is bad, it shouldn’t be practiced, if you’re given the choice.

A while ago, for a class, I wrote a blog post as an upper-middle-class white male theorizing on why women aren’t as prevalent in the Computer Sciences as men. Incidentally this also satisfied all of the requirements for my Psychology Ph.D.

Reading back through that post reminded me of this “Imagine The Possibilities” advertisement by Barbie: https://www.youtube.com/watch?v=l1vnsqbnAkk. It’s an adorable ad, until the very end where it’s revealed that Mattel is actually saying “look at all the things a girl can be! In her head, that is. While playing with her dolls, which is her rightful place. In a few years she will be a fully grown woman, and can serve Totino’s pizza rolls.”

But I digress. In the lab at university, I’ve found myself to be quite sexist when it comes to programming, I don’t know if that’s good or bad. Whenever I meet someone talking about programming, if they are a guy I will require some burden of proof that they know what they know what they’re talking about. If they’re a girl, though, I generally automatically assume they are brilliant programmers (or, have the potential to be, if they go to Degobah).

This is opposite of most traditional sexism, so I’ve never understood where the traditional sexism comes from. I still don’t. But I think I understand where mine comes from.

I’ve met a decent number of people, male and female, in the Computer Science department. The males I’ve met are from all walks of life, and all skill levels. Common is the man who knows just enough to get by, or the one who’s just here for the money but they don’t really want to be here. Sure, there are any number of brilliant male programmers, but there are any number of not brilliant ones also (I place myself somewhere in the middle).

On the other hand, the females I’ve met are the ones that stuck around through all the industry sexism, and all that’s left for me to meet are females who excel and are passionate about programming. (realize that they could have been dissuaded from a very young age, or simply weren’t given the opportunity to try, I don’t mean that the ho-hum female programmers were in a CS program and then dropped out because they felt out of place)

The end result is there are three types of people I meet: brilliant male programmers, brilliant female programmers, and ho-hum male programmers. If I see a female programmer, I automatically put them into the brilliant category, because that’s what I grew up experiencing.

Which, to be honest, is I believe where most -isms come from. Being raised in a situation where there were a lot of good apples and one bad mango. You’ll associate all mangos with that one.

But, that is my story. Take it with salt, mull it over. The way to stop sexism is to see how it works, and think about it, and discuss it.

No, that doesn’t mean comments are now an open free-for-all (though I do receive submitted comments), go start your own dang blog. Or just tell me your thoughts next time you see me.



Posted in Uncategorized | Leave a comment

Moving In Three, Two, …

I’m back from Europe, then took a short jaunt up the road to Dallas with the family before changing gears.

I spent a few weeks cranking out pull requests for cuberite, a lightweight Minecraft server written in C++ (and so avoiding the myriad problems of Java). Being essentially a fan project run by a small core group, without any support from Mojang itself, the server is perpetually behind the curve of the latest features Microsoft has released. Being a clean-room design, a lot of the weird intricacies don’t carry over from “vanilla” Minecraft – many subtle bugs that were exploited to build machines in the original don’t work here.

But, minigames (games built within Minecraft) don’t care about this sort of thing, generally. They care about things like armor and swords and hitpoints. These are often played on multi-thousand player commercial servers, where performance is the chief concern. Cuberite fits this niche exactly, in my mind.

So I’ve started a project to build a decent minigame in cuberite. Along the way I’ve been submitting PRs on GitHub (and you can, too).

If you’re into programming, it’s good to contribute to open-source projects. It helps you hone your skills, and you have fun along the way. I do it for the latter, mostly. For the former, it helps you hone the skills you don’t learn from a classroom, that are crucial for any real work. The ability to dive into a pre-existing codebase, and build a feature, I’ve found is not often taught (nor, really, can it be, not directly).

If you’re not into programming, submitting PRs just means I’m contributing my time and code to the common cause for fun. Yes, I enjoy my work.

In the midst of all of this coding, I’ve also been packing up to move out. I’ve lived with my parents my entire life (well, my Mother), now I have an apartment closer to where I’ll work (I start work in September). This entails much furniture shopping, etc., which is why this July post was posted on the 31st at 11P.M.

Busy, busy, busy. Just the way I like it.


Posted in Uncategorized | Leave a comment