← Newer posts

Older posts →

Dumb things programmers do

Posted on 2022-09-18 16:33 by randy

I’ve written before about this blog’s journey away from WordPress into becoming a statically hosted blog, and how I’ve regretted it ever since. Allow me to present you with the next chapter in this increasingly silly saga.

WordPress has a lot of limitations. It had them years and years ago when I first decided to move away from it, and it still has them now. Back then, you basically had three choices: 1. host your site through wordpress.com, meaning not a lot of customization options. 2. Host everything yourself, meaning it’s on you to keep things up to date and running. And 3. go with a hosted WordPress solution that allowed you to customize some things without having to manually manage your host or software stack. I was on option 3 at the time, and the major drawback was that it just wasn’t fast, and near-impossible to make it fast while still taking advantage of WordPress’ ease of use. Oh yeah, one constraint I forgot to mention in my quest: it had to be dirt-cheap.

So at some point I decided to statically host this blog. After all, how hard could it be, right? Just generate a bunch of files, put them in an S3 bucket and you’re done. Except you need to either use a framework that does that for you, adding a thing to learn and a depedency that could cease to be updated in the future, or do it yourself, increasing the time spent on writing and maintaining your own software. I went with the latter and wrote my own blog generation software. I tried to be strict on myself in terms of maintenance burden: I only wanted the absolute barebone functionality and as little reliance on third-party libraries as possible. I would only be satisfied if, years later, I would open up the code for this blog to add a new feature without having to trudge through thousands of lines of code, or having to stare at what I wrote in disgust and confusion.

I didn’t quite make it. Even in trying to keep the codebase minimal, it’s still a maintenance burden compared to WordPress for only a fraction of the features. I spent a long time trying to figure out how I was going to do comments on a static blog without having to rely on some shitty service like disqus. I ended up with something working, but clunky, and it only added to the maintenance burden. In retrospect I am 100% sure that staying on WordPress would have been the better solution.

So… where does a lazy programmer like me go from there? Back to WordPress? Ugh. I would have to re-convert all the blog posts I wrote in my custom format back into WordPress SQL or whatever other format WordPress allows you to import. I would have to rejig the DNS, undo all the magical things I had to do to convince AWS to host things in the way that they are.. It’s all certainly possible, but it sounds like a time sink. So instead, I did what every self-disrespecting developer would do, and I made things even worse.

One major disadvantage about this static blog is that I need to run software in order to write and publish posts. Since that software is something janky that I built myself I’m not super keen on exposing that to the world wide internet, so basically the only way I can write blog posts when I’m away is if I bring a laptop that’s got the blog software on it. Old school, I know. My solution to fix that was to come up with a little script I’d run on a local raspberry pi which would ping Github for published blogposts, and in turn render them and publish them. It would solve the problem of allowing me to write blogposts remotely, but I’d still need access to a git client.

So then I thought: why not write the blogposts on WordPress instead? I could read a WordPress RSS feed and use that to convert each post into my custom static blog format, and publish whenever there’s a new post. So that’s what I did. I now have a script that runs every couple of minutes on a local raspberry pi that checks if the WordPress blog has a new post, which it then publishes. The circle of madness is now complete. I have come full circle when really what I should have been doing is just stand still.

Eventually I will probably spend some time moving everything back to a more manageable platform. The way my life has slowed down over the years I’d say that’s probably not going to be any time soon, but let’s see how long this duct-taped solution will last, and which will be the first part to break. For now at least, I am “feature complete”. Yay.

Posted in Tech | Tagged

Change is in the air

Posted on 2022-09-09 00:41 by randy

When you’ve kept your head down for a long time you might be surprised at where you are when you finally look up again. That’s what it feels like right now.

It’s been an odd day. My life in general is pretty normal lately. As a fairly boring middle-aged person I sometimes have month after month where nothing of significance happens. Those days just fly by. But then, suddenly, something happens that jolts me from normality, and I realize fully where and when I am. Today was such a day.

First, my wife finally received her British passport. It has been a long wait for her since she first applied for citizenship, but today the last remaining worry was finally dispelled and she can travel freely again. This is amazing. One less thing to worry about.

Second, I met an old friend for dinner in London. It might be the last time I see him in a while because he’s leaving the UK, as a lot of my (software developer) friends have done in the last couple of years. Increasing costs of living in the UK make London an increasingly hard sell for a lot of people I know, so inevitably many of them are returning to their home country. The general concensus among people in my circle seems to be that London is fine for a couple of years – you earn some decent money, live the life, meet the people – but then it’s time to move on and go somewhere else so that you can have a better quality of life.

Most of my friends have left London now, either to move to places just outside of London, or further away. I’m not quite the last holdout but on days like today it certainly feels like it. It reminds me of the time when I left Japan and I was the last foreigner in my group of friends to leave. Being the last to leave is an interesting feeling. I certainly felt that same melancholy today, and it made me remember my past self.

Third, this happened:

The queen died. As I was walking to the tube station I passed by Piccadilly Circus, which was full of tourists and other random people, just standing there in a daze, staring at and taking photos of the giant billboard that was showing the queen. They all seemed a bit bewildered, wondering about what was going to happen next.

So I look up at the billboard and I realize that the world has changed so much recently. Ukraine happened. Covid happened. My life in the UK happened. And the queen is dead. Who knows what will happen next.

This is the new world
This is your time
Down in the basement
Dancing again
Everybody get ready to sing
When the lights go out
When the lights go out

Posted in Daily Life , Thoughts , UK

This is a test post

Posted on 2022-09-08 21:34 by randy

More info soon. I am working on a better way to publish blog posts. This post was published on a Raspberry Pi! Woohoo!

Posted in Thoughts | Tagged ,

The Netherlands

Posted on 2022-05-22 17:23 by randy

It's been four years since I was last back in the Netherlands. This might be the longest time that I've not been back. But covid is nearing it's end (hopefully), and circumstances allowed for it, so here I am. I didn't realise how much I missed it. A brief break from my daily life.

On a sunny spring day like this, going out for a cycle in the countryside is absolutely amazing.

Everything is green, sunny and positive. New life takes the place of old life.

Posted in Dutch , Thoughts , Travel

The electric bike

Posted on 2022-04-16 17:42 by randy

Last summer I decided to buy an electric bike conversion kit from Cyclotricity. They sell a 250W front wheel kit that doesn't require any major assembly or hacking into your bike, so I thought I'd give it a go. I'd like to think that I'm still as fit as I was ten years ago (because ten years ago I wasn't that fit either..) but I've definitely put on some weight since then.. Since my last major touring cycle I've probably put on as much as the weight of the touring luggage I used to carry, so it's as if any ride is a touring ride these days. Can't blame anyone but myself for that though.

The front wheel kit is pretty easy to install. The simplest version (the one that I got) comes with a battery, a controller box, a throttle (no pedal assist here, it's all manual control) and a front wheel that contains the motor, so in theory all you have to do is stick the bits on your bike without any disassembly other than taking the front wheel off. I did run into various difficulties though.

The first surprise was when attempting to put the new wheel onto my fork. It turns out the Surly Long Haul Trucker has some sticky-outy metal bits where the wheel connects with the fork, which is normally fine for almost any wheel because a normal wheel's wheel nuts are usually small enough to fit. But the motorized wheel's nuts were way too big, and I ended having to file large bits of my fork off so the wheel could fit. I'm pretty sure that the bits I filed off were decorative, but I really have no idea how it'll hold up in terms of structural integrity in the long term. The frame was already a write-off anyway since it was bent during a previous plane trip, so it makes for a nice experiment bike.

There were some other issues during mounting: the throttle unit doesn't fully break open, so if you want to put it into your handlebar you have to slide it on. This means having to remove the handlebar tape, brake/gear unit etc. I didn't like the idea of doing that right from the get-go, so I bought a handlebar extender instead, thinking it'd be useful for other purposes as well, like mountaing a smartphone holder and/or light unit. The one I got is way too wide though, and I'm not sure I'd buy it again. The controller unit goes underneath the frame in front of the rear wheel, and I again ran into mounting issues because the clamps provided were just a little bit too wide for my frame, and I ended up having to jury-rig that by wrapping the frame and the unit in tape to prevent the unit from jiggling around too much. It's all very ghetto, but it works.

My little bike shed is pictured below. I remember the days when I used to rent a 'studio' apartment that was a converted garage the size of this..

As you can see the cables are a bit of a mess. There's not really any way around this. I certainly could have tidied them up a bit better, but the way the package is set up is that you'll always end up with a bunch of connectors and wires that need to go somewhere. I can definitely see the value of a bicycle that has everything integrated in the frame.

I did a bunch of rides with it last year and I quite liked it. I hardly noticed the additional weight of the battery pack and wheel motor, probably because my touring bike is pretty heavy to begin with, and so am I. What I definitely did notice is the pull it provides on a hillclimb. The 'throttle' really only has one setting if you're going for the 250W model: on or off. I've cycled hillclimbs with up to a 10-12% gradient with this, and they are a piece of cake, even with a heavy bike and a heavy person on it. 250W is plenty to get up a hill at low speed with only a little human power added. If you're a hardcore cyclist and/or a glutton for punishment I'm sure it's hard to imagine 'cheating' by skipping the uphills, but for someone like me who's not in it for the performance, this is fantastic. Because of where I live I pretty much cannot avoid a 10% hillclimb if I want to go anywhere, and the motor just lets me 'skip' this without too much effort, or alternatively I can keep my speed up by adding my own power.

Before I bought this I did wonder if the front wheel unit would have enough grip when pushing, since the weight will be on the rear wheel, but so far I've not had a single issue. The front wheel motor is plenty grippy for every situation I've encountered.

What I like: it doesn't actually feel much heavier than usual, and I can skip any uphills that I don't want to do so I can preserve my strength to do longer rides.

What I don't like: the cables are an unavoidable mess, and I'm not really sure about the reliability. I did one longer ride last year where the motor kept cutting out on me despite the battery telling me it still had plenty of juice left. I couldn't reproduce that this year with a freshly charged battery, so I suspect it's not a case of faulty wiring or motor issues, but just the battery not being very accurate about its actual level. I need to do more rides to confirm that though. And even if it does fail: "escalators don't break down, they just turn into stairs". It's the same for an electric bike.

I still love cycling.

Posted in Cycling , UK

RSS Feed

Posted on 2022-04-10 16:30 by randy

Although in the previous post I lamented about wishing I'd stayed on Wordpress, a day after I published that this article about wordpress.com pricing model issues became the top post on Hacker News. Just changing out the pricing model for their existing users seems like a practice that would certainly have made me unhappy if I had been their customer. It made me feel slightly better about the time I spent on this blog. But only slightly.

And speaking of ancient technologies, since it seems I'll be on my self-built platform for a while I added support for RSS feed generation to my static blog platform. The front page now has an RSS feed, though I likely won't bother to add feeds for categories, tags or article comments unless there's more early-2010s nostalgia fans like me out there.

I'm still thinking about publishing this on Github but I know there's tons of static blog solutions out there that are way better than mine, so I've not been in a hurry. It's on my todo list though.

Posted in Tech | Tagged

Comments

Posted on 2022-04-02 15:40 by randy

I deeply regret moving away from Wordpress. It was a terrible choice. Self-hosting is much, much faster, more secure and I have more freedom to do what I want with this site, but even something as simple as making a comment form involves so many decisions and so much future maintenance burden that it just makes me sad.

Anyway, it's working. There's comments again now. I have to regenerate the entire site to make this work, but it's not something that should happen too often. I had planned to go with a full AWS-based solution but ended up incredibly frustrated because everything AWS is just so needlessly complex. Lambdas and SES seem like a great way to handle "serverless" comment form submissions, but hooking it all up and figuring out which set of permissions to use made me vow never to use AWS again if I can avoid it. Instead, pythonanywhere.com lets you very easily spin up any kind of python server, which made it super easy to get started, and the free tier provides more than enough capacity for a blog of this size. I gave up on SES as well and ended up going with mailjet which again provides a very usable free tier and a straightforward API.

I also had to add a captcha to avoid abuse. It's unfortunate, but kind of necessary on today's internet. An alternative would have been to ask people to tie an identity to their comment and authenticate them with Google, Facebook or whatever, but that runs counter to what I'd like this blog to be. If you do think this would have been better, feel free to leave a comment! You can do that now!

So in the end I had to use three new cloud services to make comments work on my static blog: a non-static web hosting service to handle the comment form submissions, a mail service to send out the comment notifications, and a captcha service to prevent abuse. In return, what I get is data that remains under my control: comments (and email addresses used in comments) are in my inbox and on my PC, rather than on some external commenting service. No ads, no spam, no bullshit. Mild hassle because of the captcha, but hey, you can't have everything.

What I would do different in the future if I still had to stick to a static/custom solution: just load everything in memory. All the 17 years of content on this blog, compiled into html, is less than 256 megabytes. Even if I became twice as prolific over the next 17 years it would still easily fit into memory. Single server, maybe with a CDN in front of it, though I doubt it'd need it. Optionally I could parse the original templates on-demand to save even more memory, but it doesn't really matter. That way I wouldn't need to jump through hoops to make static hosting work and could have everything be contained in a single web service. All of this is of course still worse than moving back to Wordpress.

Posted in Tech | Tagged

This blog as a hobby project

Posted on 2022-01-30 17:36 by randy

My rate of blogposting started going down around 2012. That's ten years ago. Between then and now I thought a lot about what I wanted to do with this blog, and if I even wanted to continue it. I probably changed my mind on that every year or so. The term 'blogging' itself has kind of become obsolete in the meantime, as did the code for the static blog generation framework I built to produce these pages. It feels weird to unironically work on 'comment forms' for a 'blogging framework' in 2022. All of a sudden it's gone from mainstream internet to a weird kind of nostalgia. Who even 'blogs' in this day and age? Plenty of people write, of course, but it doesn't take a self-hosted, self-published platform to do that. Just sign up for a Medium account, or if you don't like that there's plenty of alternatives. Some people write entire articles on Twitter, which is a whole other weird internet thing..

I can liken the experience of blogging and maintaining your personal blog software to that of driving and tinkering on a classic car. It's a fun weekend project with no practical purpose whatsoever, all commercially available solutions are way newer and better than what you're working on, and whatever you're doing will never actually be useful to anyone else. But it's fun somehow. It keeps me busy. And, to continue the analogy, there is tangible output: just like you can take a finished oldtimer back on the public road, this blog exists on the public internet. Even if it can't keep up with all the modern alternatives around it, it's still there for people to see. And that's worth something. In my case, pretty much all of my life is documented here - from all the way back in 2005 to now. I enjoy reading what I did back then, and I hope the people who knew me back then do too. If Medium is a modern-day Toyota, this blog is a poorly-put-together 80s kit car.

(I guess for most people reading this the car example is actually less relatable than that of maintaining a blog. Oh, how time flies..)

So that's what I spent my weekend on. Old man blogger. Here's what I did over the weekend, and how things easily spiral out of control..:

  • I wanted to make comments work so I had to first decide on a format and import comments from the wordpress days. This meant interacting with a wordpress SQL backup that only really restored well onto a MySQL installation, which I didn't have running, so I wrote a nasty little script that regexes over the SQL dump and extracts comments. Yum..
  • As I was playing around with the comments I noticed that a lot of the old articles' image links were broken due to the way they linked to S3 bucket content. So I did some search/replaces on historical content to fix that.
  • Because I fixed a bunch of old posts and changed the base templates I had to regenerate the full site from scratch. This turned out to be less user-friendly than I liked given the scripts I wrote a couple of years ago, so I ended up spending hours refactoring those scripts to be more user-friendly.
  • As part of the comment loading I ended up making some rapid-fire javascript changes to static files, which I had to upload manually myself every time I made a change. This eventually annoyed me so much that I updated the blog generation scripts to also be able to upload static files.
  • While working on the styling for comments I figured I might as well do some other drive-by fixes to padding, colors etc. so I improved the styling a bit.
  • The imported comments needed to be mass-published, so I added functionality to the generation scripts for that as well.
  • For the first time since starting this static blog I actually have non-broken 404 and 403 pages. omg
  • In the end I got comment display working but didn't manage to finish work on the comment submission form. I'll save that for another weekend..

It's my goal to eventually publish the static blog generation framework on github, even though I know for a fact that there's way better alternatives out there already, and I wouldn't recommend anyone to start a static blog themselves at this point as the hosted solutions are way easier. But if I can publish the code on github I can at least call it 'finished'. Perhaps when that happens I'll finally consider moving back to a hosted solution. But for now I will stubbornly keep tinkering..

Posted in Tech | Tagged

I made a Minecraft mod!

Posted on 2022-01-26 19:50 by randy

After 10 years of not touching Java I somehow made a Minecraft mod. The reason? I wanted to see if I could turn on my IoT living room light with a switch inside Minecraft. Great success!

The mod is called Breakoutbox and is available on my github page: https://github.com/rheide/breakoutbox.

Breakoutbox is a very basic mod that lets you 'break out of Minecraft' by calling external scripts on the serverside. Once you're out, anything's possible. Controlling IoT lights as mentioned above, pulling in cryptocurrency prices into your Minecraft world, pushing a button to make a Tweet, you name it. And building all those things is a hell of a lot easier in an ad-hoc external script than it is to publish a dedicated Minecraft mod every time. Anything and everything, all of the time.

It was interesting to experience the Java ecosystem again after such a long time away. Some observations:

  • The build system for Minecraft mods is really nice. The Forge MDK (mod development kit) comes with good documentation to get you started, and a gradle build file that makes it super easy to compile a mod jar or run a dev Minecraft server.
  • That said, the Minecraft code itself is closed-source, so from there on you're walking through decompiled, de-obfuscated bytecode, guessing at what things are ok to call, hoping the arguments you've provided make sense. This could've been an absolute nightmare, but thanks to the fact that Java is a strongly typed language, and community efforts at deobfuscating the code every time a new release comes out, it's surprisingly not that painful.
  • It's so difficult/terse to do simple things in Java, compared to Python. The Java standard library featureset is still a joke. No json or yaml parser in the standard lib? Bah.
  • I gave up trying to figure out how to bundle a third party library with my jar. Couldn't figure out if it's just not standard practice or if I then had to burden everyone who downloads the mod with also figuring out how to download the dependency. Either way seemed bad.
  • It's veerrry easy to get Minecraft to misbehave. The redstone behavior has a 'contract' in that running a redstone command should finish quickly, e.g. in less than .1 second. External scripts can easily take any amount of time, and I had to guard against that. I found it quite interesting how easily I could get a 'professional' game to break just by adding a few lines of code in a tiny jar file.
  • I managed to get around the long-running script issue by queueing the tasks in the Minecraft server object's internal threadpool, but I wasted hours trying to figure out why it wouldn't just work from my own threadpool. I had created a separate threadpool for my own tasks, but somehow the commands never executed if I ran them from my own threadpool. Queueing them in the threadpool that was kept in the server object fixed things. Without looking at the actual source code, which is impossible, it's very difficult to say why. My guess is that the internal threadpool had some kind of hooks attached to it that updated the game state at the end of each task, whereas with my own pool stuff just disappeared into nowhere and was ignored by the game loop. But who knows.

Anyway, it was a fun little exercise. It was a project I started in spite of wanting to finish other projects I had already started. I had no particular need for it, but having it available on github in the form of a downloadable jar that anyone can stick into their Minecraft server, and source code available for anyone who feels like forking it, does feel good. I will endeavor to finish more things this year.

(I think I finally have an idea in my head to reintroduce comments to this blog without incurring massive maintenance or financial costs. More to come.)

Posted in Games , Tech | Tagged , ,

No more analytics

Posted on 2022-01-19 20:49 by randy

Apparently Google Analytics might become illegal soon in the Netherlands. It reminded me that I still had a Google Analytics script on this blog, which I've now removed. No more tracking!

Side note: bit rot is real. I had to update various Python libs to their latest version to make this static blog platform work with the latest Python. The maintenance burden is real.

Posted in Tech | Tagged
← Newer posts

Older posts →