January 2015

How to deal with decreasing awesomeness of the internet

Posted on 2015-01-30 16:57 by rv

Is the internet well? Opinions may differ, but I'm here to evangelize mine: No, the internet is not well. The problem is with the concept of the internet as a place of freedom; a place where anything can be said and done, and shared with everyone all over the world.

That functionality, although it still exists, is becoming more and more regulated. The authorities have caught up, and certain things are starting to move to the fringes (eg. Tor hidden sites). It starts with all the blatantly illegal things, but whole-internet censorship is really not far away in many countries, including the UK, where I live.

It's not just legality issues that are making the internet less free, a far worse culprit is monetization. Youtube clips have ads on them, streaming websites throw ads in as well. No more free database service for you; the internet rule of the day is that either you pay premium, or you accept that you'll get annoyed at crap advertisements that people throw at you.

Then there's politics. Surveillance. All of these reasons make it very likely that, a few years from now, you'll no longer be able to access your favorite service for free. You don't have to agree with this, of course. Perhaps you think that the internet will be a happy shiny perfect place in 5 years, but in my opinion signs point in the other direction.

So what to do? First things first: GET YOUR DATA OUT. Never, ever, keep your only copies of things in Flickr, Facebook, Google or even Dropbox. Don't trust backup services either; they're a nice extra, but they may go bankrupt or become unreachable any time. Think of the legal issues involved: do you know exactly which files you are backing up, and do you know the rules of possessing those files in the country you are storing your backups in? I'm guessing no.

In the Netherlands it used to be legal to make a copy of any copyrighted thing, as long as it is for your own personal use and you won't commercialize it. That is no longer true, because it conflicts with European regulations which override it. Historically, this kind of thing has been hard to enforce, but the authorities have done an amazing job of catching up, and they're only going to get more in-your-face about it in the future. If you can legally justify getting something that's available in your country right now, get it now. Laws will change, availability will decrease, and you will be branded a criminal for doing something that was legal just a few years ago.

Edit: perhaps I was wrong. Google Earth was one of things I was thinking of while writing this, and I expected the free version to go away in the future. Instead, one day after I wrote this, Google made Earth Pro free. Perhaps the 'users are the product' philosophy will set us all free?

(no)

Posted in Tech , Thoughts

Gumbug: A better way to browse real estate

Posted on 2015-01-24 14:37 by rv

Last summer I really wanted to find a decent rental apartment around London. Every day I scoured Gumtree, Rightmove and the likes in search of something affordable. In the end I decided to wait until I was able to buy an apartment instead, but I spent several weeks searching and getting annoyed at real estate sites nonetheless. I decided I could save myself a lot of time and effort by automating some of the steps of my search process. My search process went roughly like this:

Go to Gumtree, search by location and price
Mentally filter out all the ads that I'd already rejected, usually because they were old or just looked crappy
Check the new ads, decide which ones I might be interested in based on my more subjective criteria (not ground floor, too far from public transport, high-crime area etc.)
Repeat the above process for a different set of locations
Repeat the above process for all locations on a different website (Rightmove, Zoopla etc.)

Thus Gumbug was born. Initally it was meant to search both Gumtree and Rightmove for rental apartments, but I've adapted it to only do Rightmove's To Buy section, for now. I've found a lot of duplication between sites that are listing property to sell, whereas for rental apartments there was often a whole category of quirky private listings that would only appear on Gumtree. The need to scrape multiple sites seems a lot less when only considering things to buy.

You can find Gumbug on github: https://github.com/rv/gumbug. I'm also running a semi-public version of it on Heroku, although it won't be very fast if a lot of people end up using it. You can have a play with it here: http://floating-forest-4090.herokuapp.com/, or to see some example search results, have a look at this link: http://floating-forest-4090.herokuapp.com/s/gzr1vwthsd. Since it might not handle the load, I'll describe how it works.

For each search you can add multiple sources, which are all consolidated into one page. I tried to avoid pagination of things as much as possible because I just want to see everything on one big page that I can scroll through at my leisure. If a listing appears on more than one source url it'll only appear once in the results. If the listing is already in the system its details won't be re-fetched every search, to save time. Adding urls as input might be a bit 'techy' but it saves a lot of coding time and allows me to specify a whole bunch of hard filters right at the source, since the url can already contain filters for price range, number of bedrooms etc.

Keywords

You can add a list of keywords to ignore and a list of keywords that are required. Eg. you can ignore 'ground floor, retirement' and you can require 'leasehold'. For the ignored keywords, if a listing contains at least one of the keywords, it'll be marked as ignored and moved to the bottom. For the required keywords, if an add doesn't contain at least one of the required keywords, it will also be marked as ignored and moved to the bottom.

Filter by distance to public transport

The public transport filter lets you select the stations you wish to be near to (or far away from). The list of stations is prepopulated from the zoned stations around London, but it'll automatically update after every search. If you add at least one station filter, all the listings will have to match at least one of your station filters, or else they will be ignored. Eg. if you add two filters: between 0.0-0.5 miles from Chesham station and between 0.2-1.0 miles from Amersham station, a listing must be either close to Chesham or close to Amersham (but not necessarily both) to match.

The distance filter is pretty stupid because distances are simply scraped from Rightmove, which (as far as I can tell) only shows straight-line distance. You might have to make a massive detour to get to the station, but Rightmove will still happily report that the listing is right next to the station.

Once the search is complete you get to see all the results on one page: all their images, important information and a map. No useless clicking through tiny thumbnails here. The key feature in the search results page is this: you can manually mark listings as either favorited or ignored, and any future searches you do from that particular search result page will preserve your favorites and ignored listings. So let's say you haven't searched anything for a week or so, all you have to do is press the search button to perform the exact same search again to get the new listings. Gumbug will pre-filter the new listings according to your criteria and will automatically move the ones you've already ignored manually down to the bottom.

So, why am I showing the ignored listings at all, if I'm clearly not interested in them? The reason for this is that humans (especially real-estate agents) make mistakes. They will mislabel things, forget to mention a keyword that every other ad that you're interested in has, or they'll add something stupid like "not ground floor" which throws off the keyword filters.

A second reason to display ignored listings is because you might be sharing the link to the search results with more than one person, and the other person might want to un-ignore a listing. Gumbug isn't exactly built on security: any person that you share the search results url with can favorite and ignore listings. This is great for me because I want to share search results with my girlfriend so she can go through them as well, but when sharing in public it's better to spawn a new search with a new url.

Lastly, there's the map. One of the things I've consistently found myself doing when checking listings, is to cross-reference the area with the deprivation map, which gives a rough indication of how much crime/poverty/incidents/bad things there are in an area. You can also click the name of each public transport station to display walking directions, so you know if that 0.6 miles is actually 0.6 miles (hint: it usually isn't).

Deprivation and Directions

Gumbug will continue to be a work-in-progress, but it's reached a point where I'm quite able to use it to make my own life easier. Maybe it can help someone else too. Here's some of its issues:

When you flag something as ignored and then go to the next page, the ignored listing will pop up again because it's been moved to the back of the sort order.
No street view support yet
Some map issues when viewing on mobile
No floor plans yet

Feel free to give it a try on Heroku. If for some reason your search doesn't seem to be working then that might be because the worker process is not running. Since Heroku's not cheap I'm running the worker process on my local machine. Heroku's database is very tiny so it might fill up very quickly. If there's enough demand I could consider setting up a more proper version of it, so consider this an attempt to gauge the public interest. Let me know what you think :)

Posted in Tech , UK

If my life was a..

Posted on 2015-01-21 23:39 by rv

If my life was a movie or a book then I'm pretty sure I'm getting close to the end credits. The initial premise, the way my character was presented at the beginning of the story, his frustrations, worries and goals, are all close to being resolved. The main storyline is nearly complete. The end credits will roll. The story will end.

And then, a new story will begin. With the same main character as the last story, but a slightly different, more evolved version. The new version of the character will have different worries, different problems, different goals. Even if the hero's journey is at an end, there will still be stories to tell. Perhaps the stories of others will take center stage in part 2. Perhaps there will be a major event in the life of the main character that will make him re-evaluate his goals. Or perhaps he'll just live happily ever after. That'd be a pretty boring story to tell, though.

Can't wait for part 2. Got to finish part 1 first..

Posted in Thoughts

Controlling foobar 2000 from Ubuntu with global hotkeys

Posted on 2015-01-20 19:21 by rv

Uh, long title, short explanation: I work primarily on an Ubuntu laptop but I listen to music from my Windows machine right next to it using foobar 2000, still the best mp3 player available (come at me bro!). On Windows I'm used to using foobar's global hotkey functionality to quickly pause and switch tracks, but on Ubuntu any way you try to pause or skip a track requires a context switch, which is damn annoying if you're in the programming zone. Here's how I solved it.

Get the foobar http control plugin: https://code.google.com/p/foo-httpcontrol/
Configure it to require a password just to be safe. Without it anyone can log in to your music player and mess up your playlist and what you're listening to. With the password on they can still do exactly that but they'll have to sniff the network packets, which really isn't worth it just to control a music player.
Once configured, use the python script below to remote-control your foobar from the shell.
Put a shell script in usr/bin (or usr/local/bin, I forget) that calls the python script with the appropriate parameter (PlayOrPause, StartNext, StartPrevious). For more commands you can check the javascript of the browser interface of the http control plugin.
In Ubuntu's hotkey configuration settings, add your hotkey and make it call the shell script you just created.

Here's the script:


import sys
import requests
from requests.auth import HTTPBasicAuth
requests.get("http://your-ip-address-goes-here:1234/default?cmd=%s" % sys.argv[1],
                 auth=HTTPBasicAuth('username', 'password'))

Voila! Cross-platform music hotkeys :D

Posted in Tech

Thoughts on buying a house in Greater London

Posted on 2015-01-16 22:51 by rv

North-West Greater London properties

Watch out for BISF houses, they're worth less.
Shared ownership is a lie. It only complicates things.
A reasonable house will set you back at least 300k, a good house 400k.
A reasonable apartment will set you back at least 250k, a good apartment 300k.
Watford station will close down soon, don't buy around there.
Check the deprivation index before you view.
Bloody hell, Chesham, that is one massive hill you got there. Check altitudes and routes to nearest public transport on Google Earth or street view.
Anything close to public transport may not have good parking spots, so check for residents parking. You'll still piss off people who are driving over to visit, though.

Estate agents

I have called at least 6 different ones so far and have been unable to arrange a single viewing in the past two weeks. Either the place is already gone or they don't call you back.
There are no deals to be found from oversight; you won't find a place that's cheaper than market price. Everything is appropriately priced. If something's cheaper than you'd expect, there's something wrong with it: close to a railway line, very old building, massive hill in the way to the nearest tube station etc.
The only 'deal' you might be able to take advantage of is getting there first. Monitor the real estate sites for new ads, but also go to estate agents in person and get them to notify you. Oftentimes they'll recommend you crap that you don't want but you can do the filtering. What's important is that you know about properties before they are advertised on the internet.
Do not trust them.

Mortgage

Holy shit.
There is a fuckton of things to research about mortgages. The more you know, the better.
Do the calculations yourself. It's the only way to be aware of what's going to happen to your money in the long term.
Check for yourself what's the difference between variable rate and fixed rate for various years.
Do the math yourself to see what a huge difference it makes if you can pay off more in the first few years.
Make sure that compound interest will not hurt you more than you can take.
Take into account that interest rates will rise, and assume a worst-case scenario.
Compared to renting and saving, getting a mortgage seems to almost always make more financial sense.
Getting a massive loan to get a really nice place is fucking stupid, don't do it. Get a more reasonable place first and then upgrade later. It'll be way cheaper. But don't trust me, do the math yourself.
Mortgage advisors (and some estate agents) are very eager for you to leave everything to them, no questions asked. Do not trust them. Ask questions. There are many lenders out there, you can always choose another.
I still cannot find any concrete info about when it is okay to renegotiate your mortgage.

WIP.

Posted in Daily Life , UK

Living responsibly

Posted on 2015-01-05 04:20 by rv

The new year is here already! I have thunk so many interesting thoughts in the past few weeks but haven't blogged about any of them. I imagine that my blog is telling me off about this in a very stereotypical grandma-like voice. If you've got a job, work. If you've got a car, drive. If you've got a blog, write. So here's some random thoughts.

It's almost 4AM right now and I have just become completely awake. The reason for this is that my girlfriend and I came back from Japan yesterday and I decided to do something I've hardly ever done lately: go to bed when I'm tired. Which, in my case, was about 9PM.

People always use the new year as a reason to make random wild claims about how one will improve one's life in the future. I tend to use the ends of holidays. Fortunately the two coincide for me every once in a while. Even if nothing comes of it, it's important to reflect and see how you can do better. A blogpost is a great way to write down what you would like to happen in the near future.

This year I only have one primary goal; one addition to life that I want to spend my energy, brainpower and money on. A house. Last summer I decided I wanted a car and a better apartment. I got a car, and searching for a better apartment made me realize that renting a more expensive one would not make sense at all in my 'grand scheme of things'. Regardless of the circumstances here, I'm realizing that a lot of things I naively thought, or wished, were variables in my life, are really constants. I tend to pride myself on being flexible and leaving my options open, particularly when it comes to where to live. But sometimes, sometimes you just have to choose a side, and reap all the benefits (and negatives) that come with it. I've already made my choice, now it's time to formalize it.

There's a lot of other things I want to do in the span of the next 12 months. Traveling. Cycling. Driving. Getting fit. Expanding my knowledge on various topics by doing some serious research and experimentation. Some of these will happen, some of them won't. There will be an inevitable lack of holiday time, willpower time or money, making it unlikely that I'll be able to do all the things that I want. I am able to say these things right now because I am fairly clear of the 'mind fog' that waitbutwhy so excellently describes. This is what good holidays do: they clear your mind and make you able to see things in the long term, accepting things that are not possible for what they are, rather than feeling frustrated about it.

This post-holiday clarity of mind is a feeling that, in my case, gets lots very easily when I go back to work again, usually within two weeks. Depending on my circumstances, I can resist the dull drone of daily life for a while, or I can choose to embrace it fully, knowing that there will be another holiday coming up in a few months. Why would I not resist it? Very simple: in daily-life-mode I am a million times more effective as a programmer. In clarity mode I think about all kinds of things, both related and unrelated to the task at hand, and I question a lot of the decisions I and others are making. In daily life mode I focus on the job at hand and don't let anything else in.

When I'm in daily life mode it's not easy for me to focus on anything else after I come home from work. Even if I leave at a sane time and don't work late I find myself not having the mental energy to delve into new things for extended periods of time. Daily-life me is displeased by this fact and kind of unhappy about it. Clarity-mode me is like "Whatever dude. It's all good, just enjoy life, man".

I've tried many coping strategies and I can't find any that work on me. The only way to deal, it seems, is to lower my expectations. If I've done a full day's work but not done anything new personally in the evening, that is perfectly acceptable. If I want to learn something new and unrelated to work, I can take time off to focus the main part of my brain on it, or, if work is not too hectic, I can accept the performance decrease at work and keep my brain in clarity mode for a while. The core point of this whole strategy is convincing my daily life self that living this way is okay. Unfortunately I need my claritic self to remind me of that, and he's not always there.

A new year, a new coping strategy. My personality changes less and less as time goes by. Some of my issues I've ironed out, others still exist. I'm getting better at being me.

There are always amazing things to look forward to. :)

Posted in Daily Life , Thoughts