Raspberry Pi controlled Lego car part 1

So, I have a fair bit of Lego lying around from my childhood days. Not a lot of technical parts, but enough to have some fun with. I also have a Raspberry Pi lying around after a failed experiment where I tried to use it as a security camera. Why not combine the two? I figured I could build a Lego car that I can control remotely via a little server on the Raspberry Pi, which would also stream its camera feed. It seemed easy enough, especially after discovering a page that explained exactly how to make the hardware part of it work. Never did I imagine it would take this much work to get it up and running..

This post will detail the hardware iterations and setbacks I faced. The next post will go into detail about the software side of things.

So, before starting on this project, here is the list of things I thought I'd need to make this work:

  • Bunch of Lego: wheels, gears, other technic parts and some generic bricks to connect it all up. (already owned)
  • Raspberry Pi plus camera module. (already owned)
  • A USB battery pack (already owned)
  • Two Lego power functions motors.
  • A 9 volt battery.
  • A motor controller.
  • Some Dupont wires to connect it all up.

Version 1

I bought the M motor starter kit and a separate M motor for my first iteration. I built a very nifty motor assembly and a very strong shell structure to house the components. Here's how it looked like:

v1.0

It is thus that I found out that I should never overengineer before testing. While building the thing I discovered that I didn't have the parts required to drive all six wheels, so I chose to just drive the middle wheels instead. I figured that by gearing down a bit it should be fine. Much nope. The M motors were way too weak to power the construction, which was already quite heavy from the USB battery pack, but weighed down even more because of its overengineered sturdiness. Having just the center wheels powered was also a terrible decision. Fail.

Version 2

v2.0

I built this is a test version to experiment with gear ratios and to see if in a lighter version the M motors would have enough power to drive the car. Answer: not really. I got it to go forward and backward pretty decently, but I still didn't have the parts required to power all the wheels, so the motors were not strong enough to turn the whole thing without a steering assembly. Fail again.

So I had two options at this point. Option one was to build a rack-and-pinion steering assembly, for which I had the parts available and did end up experiment with, but in the end I wasn't happy with its turning radius and how I'd either have to buy a servo motor or have an 'unclean' solution with a clutch gear to handle the stopping at the end of the wheel turning axis.

Instead, I went with option two: throw more money at it. I bought two XL motors and a crapload of caterpillar tracks. That'll give me the power and the grip I need to build something really nice.

Version 3

..or not. It does look impressive though. Here's a test build with the power functions battery box instead of the external 9 volt battery.

v3.0-test

At this point I was still powering the motors via a separate 9 volt battery. I figured that, since my USB battery had two USB connectors, I could use it to power the motors as well, so I ordered a USB-to-9-volt cable from Amazon. That should help me save some weight by not having to put in two batteries. Sadly, this did not go well.

USB batteries are pretty smart. So smart in fact, that they only output power if they think something's plugged in. The USB-to-9-volt cable worked fine on a powered usb hub, but every time I'd try to hook it up between the battery and the motor controller it would simply not turn on. Crap.

I found another USB battery, which is actually a battery used for starting cars that happens to have a USB port on it. This one did always keep the power on if I plugged in the USB-to-9V cable. Good! Except it only had one USB port, so I couldn't also power the Raspberry Pi from it. So I ended up with two USB batteries and over double the original weight..

v3.0-final

The big green thing is the car-charging battery. Even without the second battery the 3:1 gearing wasn't quite powerful enough to rotate this monstrosity when on certain surfaces, such as medium-thick carpet. Once again, fail. And once again, I spent way too much time overengineering the construction without testing it first with its full weight.

Version 3.5

I figured I'd try to keep the next version as small and light as possible and worry about motor powering and recharging issues later. I managed to build a pretty small prototype.

v3.5-test

Now, I had one option left to avoid the use of two batteries. The big green car charger battery did, in addition to its one USB port, come with a 12V output. It's not the 9 volts that the Lego motors need, but I did find at least one link claiming that the motors should run fine on 12 volts as well. I had previously done this as a child when connecting an old technic 9V motor to my 12V Lego train controller, which gave it quite a nice speed boost as well. I figured this would be an acceptable trick. I butchered an old power supply, cut off it's circular connector and used it to connect up the green battery's 12V port up to the motor connector.

To avoid any more gearing issues I added another axis, bringing the final ratio down to 9:1. The build ended up much larger than I had intended, but still smaller than v3.0. With better gear ratios and a higher voltage to the motors.

v3.5-final

You may notice the second battery sticking out at the end of this bulky thing, despite the fact that the green battery has a free USB port thanks to the makeshift 12V connector. This is because the stupid battery doesn't let you use both outputs at once. It just turns off one or the other. Not ideal!

I'm still annoyed at this but don't have an immediate solution. The reason I spent so much effort trying to get everything onto one battery was because I intended to build a little wall charging station, so I can drive the robot into the charging station, which will connect the charging port of the USB battery onto a USB wall socket. I gave this a try with the double-ported battery while the Raspberry Pi was connected, and sadly it resets the Pi as soon as the charging connector is plugged in. Crap times two. I haven't tried this trick with the green battery, but that's pointless anyway since it only lets you use one device at a time. Looks like I won't be able to charge the thing remotely while it's running.

V3.5 still has a whole bunch of issues. It works great in terms of motor and track performance though. It moves and turns fantastically. even loaded with two bulky batteries. The motor placement isn't great though; they're way too close to the ground. I need to fix that in the next version. It's also longer and wider than it needs to be. Once I get a solution to the two-batteries issue I'll decide on how large the next version will be. I'm also thinking about buying the servo motor and using it to power a small gearbox, but I'd also need to buy more technic gears and axles for that, and I've kind of already spent way more on this project than I thought I would spend..

For now though, I'll stick with this hardware design and focus on getting the software up and running. More on that next time.

 

Posted in Tech | Tagged ,

New hobby project

Turns out it's really easy to control Lego power functions motors via a Raspberry Pi. I found myself a new hobby project :)

2017-01-08 20.24.33

Posted in Tech | Tagged ,

The 9gag data mining scam

(Or: what is your <insert meme here> name? Click here to find out.)

Perhaps you're an occasional browser of 9gag or similar sites, and have come across images like these before:

Instagram-TropicalHouse-8dc5ec

People inevitably reply in the comments section with their hilarious name. Lately I've seen a lot more of these images than usual, and I'm guessing it's actually an attempt at data mining by some hacker trying to dox people. The questions on the images vary; sometimes it's the first letter of your name, sometimes it's the date you were born, the month you were born, the first letter of your mother's maiden name, and so on. People who are stupid enough to reply to these images with that data eventually create a very usable data trail that can be used by hackers to impersonate them on the phone. So yeah, have fun explaining to your bank that you publicly posted all details about your life in a poorly obfuscated manner.

Posted in Tech

Responsiveness and Patreon!

As you may have noticed from the blinding orange link at the top of this page, I am now on Patreon :)

That said, I am fully aware that this is a personal blog with personal content, but on the off chance that someone enjoys my writing and would like to support me, I thought Patreon provides quite a nice way to do that. I'm perfectly happy to never get anything from Patreon and will still continue blogging, as I have done for the past 11 years. Yes, it's been 11 years.. Interestingly that makes this blog the longest-lasting thing/activity/venture that I have in life. With possible exception of my iPod classic, those things last forever.

Creating a Patreon page was quite interesting. It really forced me to think further than 'It would be great if people could throw money at me for writing silly stuff like this'. I had to think about what value this blog creates, what topics are the most likely things people want to see, and what is the best way to take payment. It didn't make sense to offer payment per blogpost, since I would forfeit my freedom to write silly things that I wouldn't dare ask people money for. Monthly contribution seems to make more sense. I assumed that of my writings, the cycling trip day-by-day travel reports are probably the most interesting to read, so any funding I get on Patreon will likely go to the cycling cause.

As a secondary effect, every time I think about increasing public exposure of this blog I get nervous about its quality. I can't do much about the content after it's been written, but I can do something about the styling and user-friendliness. So yesterday I wrote some css that will hopefully make this blog a lot more mobile-friendly. It's still nowhere near perfect, but it'll have to do for now.

Random life thing: I'm supposed to be preparing for the upcoming cycling trip later this month but instead I'm currently experiencing quite possibly the worst cold I've ever had in my life. I'm feeling myself start to recover, but it'll be a few more days until I'm back to normal. I've got some ideas for posts lined up but nothing written down yet. There'll be something new here soon. Stay tuned!

(ZEST)

Posted in Tech

Why I won't be moving away from WordPress

I've made some time for myself to do some personal projects, and one of the things that's been on my mind for the longest time was to move this blog away from Wordpress and host it statically on S3 instead. Serving a single static html page (plus a few resource files, but not many) from S3 would be so much faster than letting the shared hosting server parse endless lines of php, most of which I don't even want. The benefit in that aspect is clear, but in other areas it's less obvious.

Easy of deployment is one of the issues. I'm not bothered about not being able to post by email or mobile - whenever I have something to write I'll usually have my laptop with me. It doesn't bother me that it won't be WYSIWYG either (which it wouldn't be if I write my own blog software because I don't care about WYSIWYG). The problem is always with the software and the libraries. My software of choice for my would-be static blog is python, but I'll inevitably end up requiring some libraries that will need to be downloaded and/or set up on each machine that I want to publish from. Knowing myself, I will forget to do this before I go on a trip and end up having to download those libraries at ultra-slow speed at some hotel in the middle of nowhere. Or I might be on a public machine, which will be even worse.

Comments are another issue. If the blog is completely static I'd have to go for a javascript-based comment service. I also have old comments that will either have to be converted to the new commenting system or else inserted into the html somehow if I want to preserve them. Again, there are solutions, but they're hardly easier than what I have with WordPress.

Finally there's the issue of dynamic pages: the calendar, archive and search functionality. Calendar and archive pages are pretty easily generated, but I'd either have to remove the search function or rely on an external service for that. Bleh.

The system I had in mind would be a collection of small tools:

  • A converter tool to convert a wordpress database into raw blogpost/page files (the exact format of which I would have to think about).
  • A compiler tool that reads the raw posts and writes usable html
    • Inserts content into a predefined template, I was thinking Django templates. This doesn't have to be fast.
    • Regenerates related pages (front page, archives, calenders, category pages)
    • Converts any markdown text to html
  • An uploader tool that publishes the generated html
    • Would upload to S3, perhaps pluggable so other providers would work too.
    • Would be smart enough to recognize linked images/files and upload those too.
I'd leave the template design and making the javascript commenting system work all up to the user, since that'd be a one-off job (for me, anyway). The goal of all this is to make my life easier, but it's an awful lot of work for an awfully small amount of easiness.. Anyway, if someone else would also be interested in using something like this (or willing to pay for something like this!) then do let me know.

 

Posted in Tech | Tagged , ,

The rational sacrifice

Check out this post on Elon Musk on Wait But Why. The post goes into great detail about how Musk reasons from first principle. By starting at the beginning and thinking about what's best for humanity as a whole, Musk ends up being motivated to make reality renewable energy and travel to Mars. He didn't just start there from scratch of course, and the article shows how he worked his way up from internet startups towards more lofty goals. It's a good read, highly recommended.

Somehow, when I think about my own life in this way, and trying to be as rational as possible, I do not find myself reaching the same conclusions as Musk has. I think what Musk has done is, is you consider the human factor, not the most rational solution to the problem 'what should I do with my life'. And that makes what he does all the more admirable. I'll try to explain my reasoning with some personal life examples.

In my day job as software developer it pays off to be completely and coldly rational about your product. For example, even if you intend to be on a project for 6 months, it still pays off to focus on the extreme-long-term of the project if it was made to last long, even if you don't intend to be on the project for that long. Rationally, what is best for the project is also best for you as a developer, because you are accountable for the state of the project. If the project goes well and continues to go well in the future, that means people think highly of you and will consider you again for future projects. Personal motivation and that of the whole are aligned.

Now compare that to Musk. He made his personal motivation to be the motivation of the whole. There is just no way that this can be an intrinsic, gut-feeling type of motivation, and he says so himself. He reasoned from first principles and arrived rationally at a conclusion about what he should do with his life. I know how this type of reasoning works; I use it all the time at work. In my case it means that I choose to increase test coverage, review code that someone else has already reviewed but I really want to be sure of, or spend a day debugging some hard-to-catch bug on production -- all tedious tasks that don't improve my knowledge as a developer. As a developer I can improve myself faster by learning new frameworks, trying new languages and venturing out into different areas of software. But I am employed for one particular project, and that project benefits the most from me doing what needs to be done, because no one else will do it.

I have no Musk-comparison for the opposite case: the case where you do something out of your gut feeling because you know it feels right, uncaring about the consequences, aiming towards nothing rational in particular but just wanting inner peace. But I do have an example of my own: that of cycling. Being on the road all day, seeing many things as I cycle along, with only room for one goal in my mind, sometimes pondering the book I've read the previous night. Cycling trips to me are a form of meditation, a way of clearing out my mind of unwanted thoughts and focusing on the here and now. It is a mental reset that, as far as I'm aware, I can only experience in that particular way, and it is a very powerful experience, especially when put into contrast against my daily life.

As someone who considers himself to be highly rational, I find it unreal that I am at my [best, happiest, most peaceful] when I am doing something irrational.

But then, rationally speaking, if I know that my mind and body can provide me with this peace of mind if I seek the irrational path, isn't that the path I should rationally be pursuing? And isn't everything else in my life second to that goal? It's a deeply personal (even spiritual?) goal, one that is of no use to the people around me, society or humanity as a whole. I am no Elon Musk, and I can not rationally justify me sacrificing myself to save humanity. Or perhaps I just rate my chances of success pretty low. Either way, I know I will never be spiritually motivated that way, and I think neither can he be, which is why I think of what he's doing as a sacrifice. I respect him for that, but I do not want it for myself.


Reading back what I just wrote I realized there's a contradiction between how I describe my goals versus the whole in the third and fourth paragraph. Although I wrote that I don't want to sacrifice myself in the way Musk does, it appears that I'm doing exactly that in the smaller-scale context of my project. That's probably not good. I think I can attain higher goals before resorting to a self-sacrificing position.

Posted in Tech , Thoughts

Fascinating linkdump

Yup, I'm still alive :) busy moving apartments, but otherwise still interested in THE FUTURE.

I've also recently finished a Scalable Machine Learning course at edX. It was my first time trying out an online course and it turned out to be quite interesting. Especially the final week's assignment produced some really cool results. Apache Spark is so much nicer to work with than Hadoop.

Posted in Tech , Thoughts | Tagged

A Brave New Internet (or: why I stopped using Twitter)

Twitter has been one of my favorite Internet places for at least the past 5 years. Twitter always distinguished itself from Facebook for me because of its 'just a quick thought'-ness. Anything you can think of, just dump it on Twitter. Friends might follow you on Twitter, but thoughts on there are generic and meant to be seen by the world. Anything personal that I want kept between my circle of friends goes on Facebook, anything that doesn't require a friend context goes on Twitter.

Or at least that's how I started using Twitter, but I no longer use it that way. I've been tweeting my random thoughts less and less - more on that later. My main use of Twitter for the past few years(!) has been to complain. There's nothing more satisfying when your train is running late again than to fire off an angry tweet towards the train company (that's you, TFL). Or if my mobile phone's internet has failed yet again during my daily commute (that's you, Three). Or if the software for my fitness tracker is just so shit I can't bear to use it (that's you , Garmin). But I digress. What used to be a frivolous form of quick mind-blogging has turned into an utterly useless hatefest. It's not healthy to use Twitter just for that.

So what about the other use? What about the short random thoughts? I had a fun random thought the other day at work, while performing a request for a client that was a bit out of my comfort zone, but at the same time no trouble at all and quickly handled. In a split-second I came up with "I'm a developer, not a nanny", fired off a tweet and forgot about it. My mood never darkened, I didn't brood on it, I just thought it was a funny thought because it drew parallels with the "I'm a doctor, not an X" meme from Star Trek which (I assumed) would resonate among my developer friends that are following me. Instead I got a concerned message from one of the people I work for asking me if everything was OK.

This is wrong on so many levels, but the main thing that bothers me is this: I know my colleagues quite well (I think), and I think they know me quite well. They know my personality and they know I'll speak up when something bothers me, yet somehow the off chance that an extremely generic statement I make on a personal Twitter account might somehow end up reflecting badly on my client, or my client's client, means that I will be spoken to by someone who did not understand the context in which the remark was made, or what it was even meant to represent.

And you know what the worst part is? I am in the wrong! I'm not saying that sarcastically, I truly believe that I did the wrong thing. I am wrong to assume that it's ok to share a short message publicly without context and expect people to understand all of its nuances. The past that I fondly remember is not 'better' because people back then knew you better and knew to take things in context or not place too highly a value on it; it's just that I used to get away with it because it just didn't occur to anyone to check on Twitter what people you know are saying in public. You can argue very strongly for the right to say anything you want on the internet and get away with it but from a purely game-theoretic perspective an employer would be stupid not to check. All things being equal you'd rather have an employee with zero public presence than an employee with a potentially negative web presence.

I don't fault people for thinking this way but, again from a game-theoretic perspective, my chances of remaining employed, and getting job interviews, only increases by shutting down my Twitter account. I'm not saying any public presence is a risk by definition. Github is a really good example of something that will very likely benefit you, even if you don't write a lot of code publicly. Even if you suck at coding, Github would be a representation of that. It's hard (but not impossible!) to take source code out of context, especially compared to a message of max 140 characters.

This is not 2005 any more. Back then as a fresh 20-something in Japan I could tweet and blog anything I liked without consequences. But in 2015 as a 30-something trying to be a responsible developer you simply can't blurt out random things in public.

Lack of context creates misunderstandings. I truly believe in openness of information and that, provided all parties have the full context available, more information can only have a positive effect. But no one on the internet has the time or the interest to research the full context of something before making up their opinion. That simple little fact makes Twitter a risk without a reward.

 

I've pondered on whether I should write about this at all, and I'm still pondering about closing this blog again in favor of having an anonymous blog, which is what I did a while back. I came back here, to the good old Colorful Wolf, because I believed that I could provide the context people need to understand me. I naively believe that I still can have a net positive effect on the world by writing and sharing the things that interest me.

61129068

Posted in Tech , Thoughts | Tagged ,

A tale of two bugs

A long time ago in a content management system far, far away.. a manager was sending out lifecycle emails. These emails would be sent out by the CMS automatically once a day, iterating over all recipients in the system. The CMS would weed out the recipients that did not match the email's filters, those recipents that had opted out, and those that had already received the email. It would then send the email to all the new recipients every night.

Sounds fairly simple, right? Let's complicate things a little: one of the sources of email addresses for these lifecycle emails was a customizable form that would be presented on the public-facing site. Since the email might want to reference some of the fields in the user's form submission, the email module had to be made aware of this data. Because both the email module and the forms module were unaware of each other some glue code had to be introduced: an automated task runs every night and imports form submissions from the forms module and turns them into usable recipients for the email module. For each form submission the task first calculates a checksum over the user's data; if there's already a record in the email module that matches the checksum then the same record won't be copied twice.

This system in the way that it's described here ran for many months without any issues. Until one day, just before I was about to leave of course, a project manager came to me and asked: "How come this email only has 300 form submissions but has been sent 3000 times?". A quick check of the database and server logs didn't show up any obvious bugs, so I knew I'd be working through the evening that day.

The logs confirmed that an email was indeed being sent multiple times to the same recipient. I began by trying to reproduce the problem, and immediately hit frustration. I tried to manually add myself multiple times as a recipient to a new email, but I only received it once. Then I tried to reproduce the entire workflow, creating a form on the public site and trying to register myself via there. Since the form wouldn't allow be to register multiple times, I manually added multiple form submissions for myself with the same email address. That didn't work either. Not only that: the glue code successfully detected that my multiple form submissions had the same checksum, and only created a single email recipient in the email module.

If there's one thing that sucks about programming, it's when things work when you expect them to fail. At least when things fail you've got an obvious thing you can work on, but if things are (seemingly) working you need to figure out how to break them, and that can be way more difficult.

I went back to the logs of the original email to try and find out if there was a pattern to the email addresses that were receiving the email multiple times. Something immediately stood out: out of all the addresses that the email was being sent to, only a single address received the email more than once. The real moment of clarity came when I saw the address: rather than being all lowercase, it looked something like this: "AAaaAA@BBbb.com".

With this clue I managed to reproduce the problem. I first tried to add my own email address to the email module directly using mixed case letters, but this didn't cause me to receive multiple emails. So I repeated the test doing a form submission, ran the glue code task and sent the email again. It worked! Finally I managed to reproducibly break the system.

Now I had two major clues: a) it only happens when the email address is not all-lowercase, and b) it only happens on form submissions. It was pretty easy to debug from there. Here's a step-by-step of what happened.

  • The daily email task starts up and tries to send an email to 'AAA@BBB.COM'
  • It finds that no email was sent yet, so it re-saves the email recipients with the lowercase address 'aaa@bbb.com'. It then sends the email and creates a log record confirming that the email was indeed sent to 'aaa@bbb.com'.
  • The daily data duplication task runs and happily (but mistakenly) adds a new email recipient with the address 'AAA@BBB.COM'.
  • The next day the whole process repeats itself.
Two major bugs!
  1. The email task checked if a sent record existed before sanitizing (lowercasing) the email address.
  2. Because the record in the email module got sanitized, its checksum changed. So the glue task ended up duplicating the same record over and over again every time it ran.
If there was just the data duplication bug then the email address would have gotten sanitized correctly, and only one email would ever have been sent out. If there was just the address sanitizing bug then the recipient would have been sanitized correctly after the first time, and only one email would ever have been sent out. With both bugs present things turned into an infinite loop of sanitizing and duplication.

We deleted the duplicate data, made sure that the checksum check would lowercase all fields when calculating the checksum, and modified the email task to sanitize the email address before doing the 'was-already-sent' check, not after. It would have been better to sanitize all of our existing data and to rewrite the email module, but that never happened. The entire CMS, including the email module, eventually got rewritten from scratch, but that's a story for another time.

Besides the fix, the most important lessons learned were in procedure. Always have an extra set of eyes on critical code. Code reviews are massively useful to catch problems like this. The same goes for unit testing: if we had thought of testing the email address sanitizer, it's quite likely that we would've thought of testing it with mixed case email addresses. Neither code reviews nor unit tests are an absolute guarantee that problems like this won't happen, but they make it a hell of a lot less likely.

Posted in Tech | Tagged