Gumbug: A better way to browse real estate

Last summer I really wanted to find a decent rental apartment around London. Every day I scoured Gumtree, Rightmove and the likes in search of something affordable. In the end I decided to wait until I was able to buy an apartment instead, but I spent several weeks searching and getting annoyed at real estate sites nonetheless. I decided I could save myself a lot of time and effort by automating some of the steps of my search process. My search process went roughly like this:

  •  Go to Gumtree, search by location and price
  • Mentally filter out all the ads that I’d already rejected, usually because they were old or just looked crappy
  • Check the new ads, decide which ones I might be interested in based on my more subjective criteria (not ground floor, too far from public transport, high-crime area etc.)
  • Repeat the above process for a different set of locations
  • Repeat the above process for all locations on a different website (Rightmove, Zoopla etc.)

Thus Gumbug was born. Initally it was meant to search both Gumtree and Rightmove for rental apartments, but I’ve adapted it to only do Rightmove’s To Buy section, for now. I’ve found a lot of duplication between sites that are listing property to sell, whereas for rental apartments there was often a whole category of quirky private listings that would only appear on Gumtree. The need to scrape multiple sites seems a lot less when only considering things to buy.

You can find Gumbug on github: https://github.com/rheide/gumbug. I’m also running a semi-public version of it on Heroku, although it won’t be very fast if a lot of people end up using it. You can have a play with it here: http://floating-forest-4090.herokuapp.com/, or to see some example search results, have a look at this link: http://floating-forest-4090.herokuapp.com/s/gzr1vwthsd. Since it might not handle the load, I’ll describe how it works.

For each search you can add multiple sources, which are all consolidated into one page. I tried to avoid pagination of things as much as possible because I just want to see everything on one big page that I can scroll through at my leisure. If a listing appears on more than one source url it’ll only appear once in the results. If the listing is already in the system its details won’t be re-fetched every search, to save time. Adding urls as input might be a bit ‘techy’ but it saves a lot of coding time and allows me to specify a whole bunch of hard filters right at the source, since the url can already contain filters for price range, number of bedrooms etc.

Keywords

Keywords

You can add a list of keywords to ignore and a list of keywords that are required. Eg. you can ignore ‘ground floor, retirement’ and you can require ‘leasehold’. For the ignored keywords, if a listing contains at least one of the keywords, it’ll be marked as ignored and moved to the bottom. For the required keywords, if an add doesn’t contain at least one of the required keywords, it will also be marked as ignored and moved to the bottom.

Filter by distance to public transport

Filter by distance to public transport

The public transport filter lets you select the stations you wish to be near to (or far away from). The list of stations is prepopulated from the zoned stations around London, but it’ll automatically update after every search. If you add at least one station filter, all the listings will have to match at least one of your station filters, or else they will be ignored. Eg. if you add two filters: between 0.0-0.5 miles from Chesham station and between 0.2-1.0 miles from Amersham station, a listing must be either close to Chesham or close to Amersham (but not necessarily both) to match.

The distance filter is pretty stupid because distances are simply scraped from Rightmove, which (as far as I can tell) only shows straight-line distance. You might have to make a massive detour to get to the station, but Rightmove will still happily report that the listing is right next to the station.

Once the search is complete you get to see all the results on one page: all their images, important information and a map. No useless clicking through tiny thumbnails here. The key feature in the search results page is this: you can manually mark listings as either favorited or ignored, and any future searches you do from that particular search result page will preserve your favorites and ignored listings. So let’s say you haven’t searched anything for a week or so, all you have to do is press the search button to perform the exact same search again to get the new listings. Gumbug will pre-filter the new listings according to your criteria and will automatically move the ones you’ve already ignored manually down to the bottom.

So, why am I showing the ignored listings at all, if I’m clearly not interested in them? The reason for this is that humans (especially real-estate agents) make mistakes. They will mislabel things, forget to mention a keyword that every other ad that you’re interested in has, or they’ll add something stupid like “not ground floor” which throws off the keyword filters.

A second reason to display ignored listings is because you might be sharing the link to the search results with more than one person, and the other person might want to un-ignore a listing. Gumbug isn’t exactly built on security: any person that you share the search results url with can favorite and ignore listings. This is great for me because I want to share search results with my girlfriend so she can go through them as well, but when sharing in public it’s better to spawn a new search with a new url.

Lastly, there’s the map. One of the things I’ve consistently found myself doing when checking listings, is to cross-reference the area with the deprivation map, which gives a rough indication of how much crime/poverty/incidents/bad things there are in an area. You can also click the name of each public transport station to display walking directions, so you know if that 0.6 miles is actually 0.6 miles (hint: it usually isn’t).

Deprivation and Directions

Deprivation and Directions

Gumbug will continue to be a work-in-progress, but it’s reached a point where I’m quite able to use it to make my own life easier. Maybe it can help someone else too. Here’s some of its issues:

  • When you flag something as ignored and then go to the next page, the ignored listing will pop up again because it’s been moved to the back of the sort order.
  • No street view support yet
  • Some map issues when viewing on mobile
  • No floor plans yet

Feel free to give it a try on Heroku. If for some reason your search doesn’t seem to be working then that might be because the worker process is not running. Since Heroku’s not cheap I’m running the worker process on my local machine. Heroku’s database is very tiny so it might fill up very quickly. If there’s enough demand I could consider setting up a more proper version of it, so consider this an attempt to gauge the public interest. Let me know what you think

Posted in Tech, UK | Leave a comment

If my life was a..

If my life was a movie or a book then I’m pretty sure I’m getting close to the end credits. The initial premise, the way my character was presented at the beginning of the story, his frustrations, worries and goals, are all close to being resolved. The main storyline is nearly complete. The end credits will roll. The story will end.

And then, a new story will begin. With the same main character as the last story, but a slightly different, more evolved version. The new version of the character will have different worries, different problems, different goals. Even if the hero’s journey is at an end, there will still be stories to tell. Perhaps the stories of others will take center stage in part 2. Perhaps there will be a major event in the life of the main character that will make him re-evaluate his goals. Or perhaps he’ll just live happily ever after. That’d be a pretty boring story to tell, though.

Can’t wait for part 2. Got to finish part 1 first..

Posted in Thoughts | Leave a comment

Controlling foobar 2000 from Ubuntu with global hotkeys

Uh, long title, short explanation: I work primarily on an Ubuntu laptop but I listen to music from my Windows machine right next to it using foobar 2000, still the best mp3 player available (come at me bro!). On Windows I’m used to using foobar’s global hotkey functionality to quickly pause and switch tracks, but on Ubuntu any way you try to pause or skip a track requires a context switch, which is damn annoying if you’re in the programming zone. Here’s how I solved it.

  • Get the foobar http control plugin: https://code.google.com/p/foo-httpcontrol/
  • Configure it to require a password just to be safe. Without it anyone can log in to your music player and mess up your playlist and what you’re listening to. With the password on they can still do exactly that but they’ll have to sniff the network packets, which really isn’t worth it just to control a music player.
  • Once configured, use the python script below to remote-control your foobar from the shell.
  • Put a shell script in usr/bin (or usr/local/bin, I forget) that calls the python script with the appropriate parameter (PlayOrPause, StartNext, StartPrevious). For more commands you can check the javascript of the browser interface of the http control plugin.
  • In Ubuntu’s hotkey configuration settings, add your hotkey and make it call the shell script you just created.

Here’s the script:


import sys
import requests
from requests.auth import HTTPBasicAuth
requests.get("http://your-ip-address-goes-here:1234/default?cmd=%s" % sys.argv[1],
                 auth=HTTPBasicAuth('username', 'password'))


Voila! Cross-platform music hotkeys

Posted in Tech | Leave a comment

Thoughts on buying a house in Greater London

North-West Greater London properties

  • Watch out for BISF houses, they’re worth less.
  • Shared ownership is a lie. It only complicates things.
  • A reasonable house will set you back at least 300k, a good house 400k.
  • A reasonable apartment will set you back at least 250k, a good apartment 300k.
  • Watford station will close down soon, don’t buy around there.
  • Check the deprivation index before you view.
  • Bloody hell, Chesham, that is one massive hill you got there. Check altitudes and routes to nearest public transport on Google Earth or street view.
  • Anything close to public transport may not have good parking spots, so check for residents parking. You’ll still piss off people who are driving over to visit, though.

Estate agents

  • I have called at least 6 different ones so far and have been unable to arrange a single viewing in the past two weeks. Either the place is already gone or they don’t call you back.
  • There are no deals to be found from oversight; you won’t find a place that’s cheaper than market price. Everything is appropriately priced. If something’s cheaper than you’d expect, there’s something wrong with it: close to a railway line, very old building, massive hill in the way to the nearest tube station etc.
  • The only ‘deal’ you might be able to take advantage of is getting there first. Monitor the real estate sites for new ads, but also go to estate agents in person and get them to notify you. Oftentimes they’ll recommend you crap that you don’t want but you can do the filtering. What’s important is that you know about properties before they are advertised on the internet.
  • Do not trust them.

Mortgage

  • Holy shit.
  • There is a fuckton of things to research about mortgages. The more you know, the better.
  • Do the calculations yourself. It’s the only way to be aware of what’s going to happen to your money in the long term.
  • Check for yourself what’s the difference between variable rate and fixed rate for various years.
  • Do the math yourself to see what a huge difference it makes if you can pay off more in the first few years.
  • Make sure that compound interest will not hurt you more than you can take.
  • Take into account that interest rates will rise, and assume a worst-case scenario.
  • Compared to renting and saving, getting a mortgage seems to almost always make more financial sense.
  • Getting a massive loan to get a really nice place is fucking stupid, don’t do it. Get a more reasonable place first and then upgrade later. It’ll be way cheaper. But don’t trust me, do the math yourself.
  • Mortgage advisors (and  some estate agents) are very eager for you to leave everything to them, no questions asked. Do not trust them. Ask questions. There are many lenders out there, you can always choose another.
  • I still cannot find any concrete info about when it is okay to renegotiate your mortgage.

WIP.

Posted in Daily Life, UK | Leave a comment

Living responsibly

The new year is here already! I have thunk so many interesting thoughts in the past few weeks but haven’t blogged about any of them. I imagine that my blog is telling me off about this in a very stereotypical grandma-like voice. If you’ve got a job, work. If you’ve got a car, drive. If you’ve got a blog, write. So here’s some random thoughts.

It’s almost 4AM right now and I have just become completely awake. The reason for this is that my girlfriend and I came back from Japan yesterday and I decided to do something I’ve hardly ever done lately: go to bed when I’m tired. Which, in my case, was about 9PM.

People always use the new year as a reason to make random wild claims about how one will improve one’s life in the future. I tend to use the ends of holidays. Fortunately the two coincide for me every once in a while. Even if nothing comes of it, it’s important to reflect and see how you can do better. A blogpost is a great way to write down what you would like to happen in the near future.

This year I only have one primary goal; one addition to life that I want to spend my energy, brainpower and money on. A house. Last summer I decided I wanted a car and a better apartment. I got a car, and searching for a better apartment made me realize that renting a more expensive one would not make sense at all in my ‘grand scheme of things’. Regardless of the circumstances here, I’m realizing that a lot of things I naively thought, or wished, were variables in my life, are really constants. I tend to pride myself on being flexible and leaving my options open, particularly when it comes to where to live. But sometimes, sometimes you just have to choose a side, and reap all the benefits (and negatives) that come with it. I’ve already made my choice, now it’s time to formalize it.

There’s a lot of other things I want to do in the span of the next 12 months. Traveling. Cycling. Driving. Getting fit. Expanding my knowledge on various topics by doing some serious research and experimentation. Some of these will happen, some of them won’t. There will be an inevitable lack of holiday time, willpower time or money, making it unlikely that I’ll be able to do all the things that I want. I am able to say these things right now because I am fairly clear of the ‘mind fog‘ that waitbutwhy so excellently describes. This is what good holidays do: they clear your mind and make you able to see things in the long term, accepting things that are not possible for what they are, rather than feeling frustrated about it.

This post-holiday clarity of mind is a feeling that, in my case, gets lots very easily when I go back to work again, usually within two weeks. Depending on my circumstances, I can resist the dull drone of daily life for a while, or I can choose to embrace it fully, knowing that there will be another holiday coming up in a few months. Why would I not resist it? Very simple: in daily-life-mode I am a million times more effective as a programmer. In clarity mode I think about all kinds of things, both related and unrelated to the task at hand, and I question a lot of the decisions I and others are making. In daily life mode I focus on the job at hand and don’t let anything else in.

When I’m in daily life mode it’s not easy for me to focus on anything else after I come home from work. Even if I leave at a sane time and don’t work late I find myself not having the mental energy to delve into new things for extended periods of time. Daily-life me is displeased by this fact and kind of unhappy about it. Clarity-mode me is like “Whatever dude. It’s all good, just enjoy life, man”.

I’ve tried many coping strategies and I can’t find any that work on me. The only way to deal, it seems, is to lower my expectations. If I’ve done a full day’s work but not done anything new personally in the evening, that is perfectly acceptable. If I want to learn something new and unrelated to work, I can take time off to focus the main part of my brain on it, or, if work is not too hectic, I can accept the performance decrease at work and keep my brain in clarity mode for a while. The core point of this whole strategy is convincing my daily life self that living this way is okay. Unfortunately I need my claritic self to remind me of that, and he’s not always there.

A new year, a new coping strategy. My personality changes less and less as time goes by. Some of my issues I’ve ironed out, others still exist. I’m getting better at being me.

There are always amazing things to look forward to.

Posted in Daily Life, Thoughts | Leave a comment

Traveling memories

It’s odd how the act of traveling can make you remember things so much more vividly. Just sitting in the plane to Japan I remembered vividly my last cycling trip, and the many before it. My memories seem to more strongly associated with places than with people. Maybe that’s because the people end up migrating elsewhere? Or maybe it’s because people change more rapidly over time than places do? With places I focus on similarities compared to the past, whereas with people I can’t help but notice their differences.

Next year it will have been 10 years since I first went to Japan and changed my life forever. It’s when I made the most severe mental change in my whole life so far. There have been many small, incremental improvements to ‘the concept of me’ over time, but Japan was definitely a version 2.0. Hopefully there weren’t too many regressions.. I need a testing framework for my personality..

I’m currently sitting in the domestic terminal of Narita, waiting for a flight to Naha. It’s been a long time since I’ve been there, yet it feels like going back to somewhere familiar and safe. It’s not quite like feeling at home, yet close to it. It’s more like a feeling of unguardedness and a ridiculously high (over)confidence that nothing bad can happen, or at least that I’m prepared for all bad things that might happen. Just like every other time I went to Japan in the last 7 years, that feeling starts right after I board the plane at Heathrow or Amsterdam. It’s not an end-goal kind of feeling -the feeling alone is not enough to satisfy the mind- but it provides a foundation from which to start doing more extreme things. The only other place I’ve ever been to that felt the same was the Seychelles.

No goal for this blogpost, except perhaps to collect a few loose thoughts before the year ends. I’m still learning how to act my age; my mental age seems to age less fast than my physical age. But I feel that I’m catching up. And the more I catch up, the more clear my future life decisions become. 2014 was a fantastic year. 2015 will be even better. There are still countless achievements in life left to be unlocked.

Posted in Thoughts, Travel | Leave a comment

You can’t always get what you want?

Sometimes I buy something nice for myself. Something that is not the best-value-for-money product, but something a bit more expensive. It’s better, of course, just not quite as betterer as you might expect from the price tag. Yet, unquestionably, it’s better. I enjoy such a purchase greatly because I know I bought something good, but I also feel guilty about having spent money when I didn’t really have to. After such a purchase I feel like I need to stop spending for a while, so even if there’s something else that I really want, I’ll delay buying just for the sake of it, even though I can afford it.

This seems to happen to me with complete disregard to price range. I’ll feel bad about going for the slightly more expensive meal (~20GBP), the slightly more expensive Lego set (~50GBP), the slightly more expensive headphones (~200GBP) or the slightly more expensive bike (~1000GBP). Just reading back what I just wrote makes me feel guilty for buying all those things, even though I know I can afford them and enjoy all of them.

As you get older there’s more and more purchases coming up: a car, a house, a hotel, a planet, a galaxy. I feel the need to have worked harder for each thing that I own because I taught myself that if you want to buy something, you should suffer. The idea of “I just bought that and didn’t even break a sweat” implies a never-ending guilt trip to me. But only if you buy nice things. Buying crap things is OK. You don’t feel the need to suffer so much when you’re buying crap things. The problem with that is, you’re buying crap things.

Perhaps I just like tormenting myself. I’d just really like it if I could be happy about the nice things that I buy without feeling guilty about it..

——–

I was going to call this post “You can always get what you want” and make it about how I can afford the nice things instead of the average things, but a) the tone was very dickish and b) I didn’t actually get what I want cause I don’t want a guilt trip and I have one. Damn you, Rolling Stones, you were right after all.

 

Posted in Thoughts | Leave a comment

The GT86 goes to Assen TT track

So, an odd weekend, and the chance presented itself to try out the new braking system on the GT86 before the cold winter. Or rather, during. After a long but satisfying drive from Calais all the way to the north of the Netherlands, I made it to Assen, the home of the Dutch motor grand prix. Cold, gloomy weather awaits.

That, and chavs. Lots of Dutch chavs, or ‘sjonnies’ as the Dutch prefer to call them. Lots of 90’s and 00’s BMWs, Golfs and the like. In the UK you had your fair share of chavs, but also a reasonable number of ‘car enthusiasts’. It’s hard to write down the difference between the casual chav and the car enthusiast, but when you see it, you’ll know without fail.

The organization, done by vrijrijden.nl, was casual and free. Signup was easy and there was no hassle with sound checks at all, but you do get a transponder to carry around in your car. Before each session the cars do a pre-line-up at the entrance to the pit lane, and then line up at the pit exit to leave in single file. There is no safety briefing and there are no rules about overtaking, using the kerbstones or drifting. A huge difference compared to the strictly-managed UK tracks.

All this freedom sounds nice in theory, but is definitely not when you’re racing around with a bunch of chavs in beat-up cars with a death wish. I’ve seen many people spin out, run off track, blow up their engines or have some other mechanical failure. There were loads of individual driver errors, way more than I’ve seen on Snetterton or Bedford. Nearing the end of one session, about three cars spun off or broke down in a single lap and the session was suspended. Lastly, because there were no rules about where not to to overtake, some cars drove needlessly competitive and took risks that really weren’t justified given the situation. A particular orange 350Z springs to mind: every time that guy overtook someone it came to a near-crash. A total disregard for other drivers. He was a fast driver, but an idiot who had no regard for the situation.

Ok, that’s all the bad stuff out of the way. Now for the good stuff! The track was amazingly fun to drive after the cars had dispersed a little. Loads of fast, technical corners with some good hard braking points as well. I had some great battles with various cars, including a pimped-up Mini Cooper that I just barely could not catch before the session ended. Another highlight for me was holding off a deceptively beat-up 90’s Honda CRX with a very skilled driver who kept catching up with me in the corners. I really had to drive all out to keep in front of him.

Things didn’t go super-smoothly for me either. In the double right handers before the final straight I nearly lost my car twice. The first time I just overshot the corner like an idiot because I judged my speed wrong, but I somehow managed to hold it onto the runoff area and get back on track. The second time I pushed myself a little too hard and swung the rear out a bit, but again managed to hold it.

The worst incident I had was in the very first corner, on cold-ish tires, right after the second session had begun. I’d done a lap already so assumed the tires were ready, and then tried to push a little to pass a car in front of me, but carried way too much speed into the corner. Because I had so much speed and the angle was so very wrong, I tried to bleed the speed rather than try to drift my way through it, but I overdid it with the steering and caused the rear to break out (which is quite an easy thing to do in that corner). To make matters worse, just when I thought I had caught it with a nice correction, I slid off track over a bump into the runoff area, causing an even worse angle. In the end I did manage to catch that as well, but it certainly wasn’t graceful.

The GT86 is ridiculously controllable, but also quick to hit its limits. It seems to have two modes: either you’re stable turning into a corner, meaning you’ll end up in understeer if you push it. But if you use the brakes right, or if the track induces the right kind of bumps, then the rear gets light and jumpy, which is exactly what you’ll want it to do. Then you can choose to straighten out a bit to accelerate smoothly and quick, or dab the throttle or steering just a little more to induce a lovely little slide. It’s just so entertaining to drive. I hardly ever found myself frustrated at how it handled, right up until the end of the third session when the tires started to feel a bit worn out, causing a lot more understeer.

The brakes are an interesting story. I’ve had the stock discs and pads replaced for a set of DBA 4000 slotted discs and Hawk brake pads. I also had the brake fluid changed for something that can handle higher temperatures. I was expecting the brake performance to be at least as good as before, and hopefully without as much fade, jittering or wear as I experienced on the original ones. To be honest, I can’t say that I’m flabbergasted by their performance. The braking force was perhaps equal to the original brakes at the start, but it definitely got worse during the second session. It did remain stable after that and did not deteriorate at all any more, so they’re definitely longer-lasting than the originals. Braking at full force was only barely able to trigger the ABS to kick in on a straight line, which I think means that the tires can hold their grip at the maximum amount of force the brakes are able to exert on them.

I dunno. They just feel weaker. In the end I felt more confident that I wouldn’t have shitty side effects from hard braking, but I also knew that I’d have to pick my braking point early, and that I wouldn’t be able to brake just that extra bit harder if I really needed to. Opinion inconclusive. I’ll have to try them again at the next track day.

The brake pedal travel did increase during the session, again up until a point where it remained stable. It now catches at a fairly lower point. At first I thought that it was because the fluid had boiled causing the gas in the lines to compress whenever I brake, but now I’m wondering if brake pad wear isn’t a more simple explanation. If the pads are just that much thinner I just have to push that much more in order for them to reach the discs. That would mean that I’ve severely underestimated the lifetime of brake pads, though, and would probably need a new set after another 1 or 2 track days. Not good..

Assen was an awesome track and very memorable. I’d rank it just above Snetterton and easily above Bedford in terms of excitement and technical challenge level. But I would not want to drive it again with vrijrijden.nl, simply because there’s too many unskilled drivers in dangerous cars. There really should be a ‘non-chav’ class at these kind of events..

Right. Time for winter.

Posted in Cars | Tagged , | Leave a comment
← Sidebar