Thoughts of a geek

4 October 2014

Geotagging photos with Google Location History

Filed under: Computers — Tags: , , , — qwandor @ 12:14 pm

When I take photos on my Nexus 5, it automatically geotags them, which is great. However, when I am on holiday, I often use a proper camera with a bigger sensor and lens as it can produce better results. The disadvantage of this is that it does not have a GPS so cannot geotag the photos I take. Fortunately, I have Google Location History enabled on my phone, which records my location periodically. I use digiKam to manage my photos, which has a feature to correlate photos with a GPS track by timestamp. Here is how I use this to geotag my photos:

  1. Enable location reporting and location history on phone. In Android this is under Settings→Location→Google Location Reporting.
  2. Ensure camera clock is correct.
  3. Take photos.
  4. Go to the Google Location history web interface, select the relevant days, and click Export to KML.
  5. Convert the KML file to GPX with this GPS Visualizer tool. Unfortunately the converter has a bug: the KML file has a timezone of UTC-7:00 for all timestamps, but the converter ignores this and treats them all as UTC. We will have to compensate for this in the next step…
  6. Select the relevant photos in digiKam, and click Image→Geo-location from the menu. Load the GPX file you downloaded in the previous step. To compensate for the bug mentioned above, you will need to set the camera time zone option to 7 hours ahead of the actual timezone your camera was set to. So if you were taking photos in UTC+2:00, then set it to UTC+9:00.
  7. Click Correlate. Check that your photo geo-locations look correct. If they look wrong, ensure that you got the timezone right. If they look right but some are missing, try enabling the Interpolate option.
  8. Apply and Close. Done!

4 October 2010

More requests

Filed under: Computers, Electronics, Me, music, thequotebook — Tags: , , , , , — qwandor @ 2:34 am

Continuing on from my last post, here are some more topics people requested me to write about.

xyzzy and Craig both want me to write about working at Google. Hmm. Well it is a fairly good place to work. We get free breakfast, lunch and dinner in the office, Monday–Friday, plus there are microkitchens around the place with snacks and drinks. The food is pretty good too. Typically at lunch in the main cafeteria there will be 2 or 3 choices of meat (at various times we have had roast duck, roast beef or lamb, lamb cutlets, beef steak, various kinds of sausages, sausages wrapped in bacon, chicken, turkey, I think there was even kangaroo steak one time), some choice of fish or other seafood, vegetarian dishes, roast and/or boiled potatoes, rice, a range of cooked vegetables, various salads, sushi, a couple of flavours of soup, bread of various sorts, crackers and cheeses, a hot pudding, cakes or cupcakes or other cold desserts, fruit, and a range of drinks. There is a machine to squeeze oranges into juice which is fun to watch. There is not nearly so much selection for dinner unfortunately, and not so many people stay for it. Breakfast is pretty good though. If you happen to be in London, come for lunch! We get to have up to two social guests to eat at the office each month.

Food aside, the work has been fairly decent. It does take a while to get used to all the systems and technologies, as Google have a lot of internally-developed tools and systems that nobody else uses, and have had to put a lot of work into making everything scale bigger than pretty much anyone else. Writing a web application that could cope with being used by everyone on the Internet, many simultaneously, from all over the world, does pose some challenges over just designing something for a few hundred people. I still have a lot to learn. Unfortunately nothing I have worked on so far has been launched yet, and it is unlikely that any substantial part of it will be launched publicly until sometime next year.

Working at a global company with tens of thousands of employees is certainly rather different to working at a small place with between 10 and 20. The ratio of actual developers to support roles (legal, HR, recruitment, management, sales, facilities, and probably many others that I do not even know about) seems to be quite a bit lower. I am not too sure what differences can be attributed to differences between England and NZ as I have only worked at the one company over here. I guess there are more big companies over here though, being a bigger country. I do find that there is not so much chance to chat with my colleagues about anything outside work, though that may be at least partly due to the fact that I knew about half of the guys at Innaworks already before I started working there, though university and Interface.

As for what skills I use from my studies, the data structures, algorithms and complexity stuff was certainly important in passing the interviews and getting the job, though so far at least it has not been all that important day to day.

I think that answers most of Craig’s questions. xyzzy also wanted to know about “some project you’ve embarked upon in your spare time! your take on what really matters in life!”. Unfortunately I have not been working on any terribly exciting projects in my spare time since I moved here. I did just recently order a Bus Pirate, in the hopes of hooking some sort of I2C devices up to my computer and having a play. That has not yet arrived yet though. It is also not clear whether there are proper drivers for it to let it work with the usual Linux I2C infrastructure, or just userland stuff.

I have made a number of updates to theQuotebook in the last month or so, most notably finally getting the new style ready to launch. I am working on a new feature to allow quotes to be added by email, and Twitter support may also follow if I can work out a good way to design it and find a suitable Twitter username that is not already taken.

I also did some work on a web-based client for Fridge a while ago. It is implemented using GWT, and talks directly to the XML-RPC API that the other clients use, so it supports all the same features, including Interfridge. You can try it out if you have an account on the Memphis fridge or any peered fridge.

So what really matters in life? Probably not Slayer. God, I think, though I am not sure quite how. People seem to be fairly important. I wish I got to spend more time interacting with people constructively, perhaps sociably as well. Finding good community and good friends is hard. Knowledge is also important I think. Do more maths, people. And some science if you like.

Before I stop for the now, Richard asked me (twice!) to write about Slayer, so I guess I should do that. Richard seems to be a little bit obsessed with Slayer, though I am sure he would assure you it is purely in an ironic way. I am afraid I do not now much at all about them, so I will be looking up what I can. last.fm informs me that Slayer are “a thrash metal band from US, formed in 1981 … credited as one of the ‘Big Four’ thrash metal bands, along with Metallica, Anthrax, and Megadeth”. They seem to choose deliberately controversial topics for their songs, which is not uncommon among such bands. I am pretty sure Metallica are better. Perhaps you can tell us more, Richard?

I think that will do for today. Next up: books, culture and language if I can think of anything to write about them, and probably not a vat of hot chocolate.

18 February 2010

Your fare is Mifare

Filed under: Computers, University — Tags: , , , , , — qwandor @ 9:22 pm

Or rather, your student ID.

I recently bought a Snapper Feeder to have a play with, as I had heard that they were supported by libnfc under all the major operating systems and could talk to a variety of contactless smartcards. I tried any cards I could get my hands on, and other than the Snapper card itself I found that our building access cards were not compatible (using a different frequency, perhaps?) but the Vic student ID cards were recognised. Furthermore, it turns out that these ID cards are MIFARE Classic 4k cards.

Now, this is interesting because encryption scheme used by the Mifare Classic was broken and research published explaining the vulnerabilities as early as 2007, and this is even acknowledged by the manufacturer. Anyway, being the curious sort I am, I went about trying to see what I could see about the Vic student ID cards. I have not yet been able to replicate the card-only crack to recover the keys, but I have had a bit of a look at how the cards are formatted.

Firstly, a bit of background. The Mifare Classic 4k has 40 sectors, each of which has 2 48-bit encryption keys (called A and B) and 12 configuration bits which control which of the keys allow read and write access to the sector’s data and configuration. Each sector is broken down into a number of 16 byte blocks. The first 32 sectors have 4 such blocks (64 bytes total), while the last 8 sectors each have 16 blocks (256 bytes per sector). The last block of each sector is called the sector trailer and contains the encryption keys and configuration bits previously mentioned. Note that the configuration and keys for each sector is independent of all the other sectors. Reading from and writing to the card is done on a block by block basis. Accessing a block is a two step process. First you must authenticate to the sector with either the A or the B key, then you can read or write one of the blocks in that sector.

With this in mind, here is what I have found so far about Vic’s student ID cards. I used the micmd tool, which provides a fairly simple interface to access Mifare Classic cards using libnfc, and a few other bits and pieces. Authentication to all sectors except sector 15 worked using FFFFFFFFFFFF (a common default key) as either key A or key B. However, despite the successful authentication, I was only able to read the blocks of sector 0. (Admittedly I did not try all of the other sectors, but all that I did try failed to read.) This may indicate that these sectors are configured to not to be accessible by either key, as a way of permanently disabling them, or it may be a problem with my reader. The reason I suspect my reader is that it would often lock up after certain operations, not responding at all until I unplugged it and plugged it back in again. I am not sure what is causing this; if anyone has any ideas do say.

Sector 15 appears to be using a proper key, and is probably where the real data of the card is stored.

Sector 0, the one sector I did manage to successfully read, does not appear to hold much of interest. On one card, with a UID of D4 EE 01 6E, the four blocks were

0: D4EE016E55980200648E565165603905
1: 800F0000000000000000000000000000
2: 00000000000000000000000000001248
3: 000000000000787788C1000000000000

Block 0 of sector 0 apparently holds read-only data set by the manufacturer, so is not that exciting. It appears that the first 4 bytes are the UID of the card. The 5th byte also seems to vary between cards (on the 3 cards I tried the values were 0x55, 0x5F and 0x61). The remaining 11 bytes of block 0 were the same on all 3 cards I tried. Blocks 1, 2 and 3 (the trailer block) were also the same on all 3 cards, which suggests that they are unlikely to be interesting.

I did attempt to use the nested authentication attach (I believe) as implemented by mfoc and MFCUK to recover the keys for sector 15, but for some reason both implementations failed, possibly due to the reader ceasing to respond part-way through as mentioned above. Any suggestions on how to get past this are welcomed.

Does anyone have any other interesting smartcards?

4 November 2009

Rubgy, a safe technical topic, and baking

Filed under: Computers, Me, Recipes — Tags: , , , , , , — qwandor @ 10:33 pm

Continuing on from my last post taking suggestions from Twitter, today I will be blogging about topics suggested by people on Facebook.

Allan Chesswas: Rugby

Here in New Zealand, and elsewhere I hear, some people like to run around a field jumping on each other and chasing an oddly-shaped ball. This activity seems to be more popular among boys than girls. Many other people like to watch them do so. Several of my flatmates are in the latter set, and one in the former. I have never seen the attraction of either activity. The running around does at least provide decent exercise, but at significant risk of injury. Watching seems even more pointless, and frequently people do so (via television) instead of partaking in more interesting and productive pastimes such as good conversation, programming, baking, listening to music or even reading. This can be something of a frustration.

Donald Gordon: Safe technical topics which no one will be offended by.

Well. Most of my ‘recreational’ programming of late has been on Fridge. Fridge is a co-operative honesty system for snackfood which originated in Memphis. The Memphis Fridge is something of an open secret, used by graduate students and their friends. The basic idea is that certain people buy pre-packaged food and drink in bulk. Everyone who uses fridge has an prepay account which they can put money into by putting cash in a drawer and crediting their account, and then they can use this balance to buy the food and drink. The software keeps track of users, money, the various items stocked, markups to make up for lost or damaged stock, various statistics, and so on.

I have made a few minor contributions to the Fridge software in the past (such as adding a QIF export feature to allow users to import their transaction history into their personal accounting software), but my substantial work on it now began with interfridge. Interfridge was an idea that I and others had some time ago, last year I think, to allow users on one fridge to use that account on another fridge. This was motivated by the fact that both Memphis and Innaworks (where I currently work) run the Fridge software, and several of us use both fridges from time to time. The idea of interfridge was that I could visit Memphis, login to their fridge with my Innaworks fridge account, and purchase items from Memphis just like any Memphis user.

At some point Donald wrote some notes on his ideas of how interfridge should be designed. I finally got around to implementing this (the server side in PHP as part of fridgeweb, the client side integrated into the Java fridge client) using a protocol on top of HTTP. After much discussion with lorne, Chris Andreae and Stephen Nelson and several iterations changing the protocol to fix potential security holes pointed out, we came up with the interfridge protocol that is now in use between Memphis and Innaworks.

However, we realised along the way that fridge really could do with being improved in other respects. Currently, the Java client talks directly to a PostgreSQL database which holds all the user, stock and transaction data. This means that the client must include the database password, which is really not a good thing to be giving out to all and sundry. A much better design would be to have a trusted fridge server which talks to the database, and then have the client perform all operations via the server. With an appropriately-designed protocol, this means that the client need not be trusted, and so anyone can write their own client to use with the fridge. I have thus begun to design and implement a fridge protocol based on the interfridge protocol (it can do everything the interfridge protocol can do and more, so will replace it when it is done). For this I scrapped the custom RPC protocol which I had been implementing for myself on top of HTTP, and went with XMLRPC to take care of all the details of procedure calls, encoding and decoding in a standard way. There are various implementation of XMLRPC available for many programming languages. I should note that the documentation linked above does not include all of the API currently implemented in the code, though it will eventually. I am currently still in the middle of designing it, implementing and documenting it at the same time.

As it currently stands, the fridge server has methods to login, check a user’s balance, transfer an amount to a local or remote (i.e. interfridge) user, make a purchase at the local fridge (I still have not yet decided the best way to handle interfridge purchases in the new scheme), list the current stock, and list other fridges with which the fridge is peered. I am writing a client library and simple command-line client in Ruby to test this as I go. The command-line client implements all the features just mentioned. I have not yet done much on the Java client, but that will come at some point. lorne has just started work on an iPhone client, which should be cool.

Felix Shi: You can always blog about cooking weird and wonderful dishes 😀

Weird and wonderful? Not sure about that. I made a batch of muesli and a chocolate cake on Sunday, as I mentioned. The weekend before I baked two batches of muffins (apple and sultana for the Skyline walk on Saturday, banana chocolate chip or something for a picnic on Labour Day with gringer, ja and xyzzy).

I am afraid my cooking of late has not been terribly interesting. Suggestions are welcomed, as are visitors interested in consuming baking. As always.

Fixing VMware mouse grab bug on Ubuntu Karmic

Filed under: Computers — Tags: , , , — qwandor @ 9:23 am

I just upgraded my work machine from Ubuntu 9.04 (Jaunty) to 9.10 (Karmic) and came across a couple of problems with VMware Player (version 2.5.3 build-185404). It seemed wise to document the fixes I found here so that other people experiencing the same problems might find these solutions when they Google for it.

The first was an error when it tried to launch my VM complaining that the virtualisation extensions of my CPU were already in use, saying “The virtualization capability of your processor is already in use. Disable any other running hypervisors before running VMware Player.” and then a number of other errors. This was fixed by removing the KVM kernel modules:
$ rmmod kvm_intel kvm

The second problem was that the VMware window would lose its mouse capture whenever I moved the mouse pointer outside the top-left of the VM screen (apparently a 640×480 region), unless I had a mouse button held down. This made it impossible to actually use the VM. This was eventually fixed by adding the following line to /etc/vmware/bootstrap:
export VMWARE_USE_SHIPPED_GTK=yes
This forces VMware Player to use its own version of the GTK library rather than the Ubuntu one, which apparently avoids the mouse grab bug.

8 September 2009

An idea for songfighting

Filed under: Computers, music — Tags: , , , , , — qwandor @ 11:14 pm

I was walking to work yesterday, and I had an idea.

Perhaps you have heard of SONG FIGHT. If not, the idea is fairly simple: each week, a song title is given. Competitors then have a week to write the words and music for a song fitting that title, record it, and submit it. Anyone can then download the songs, listen to them and vote for which they think are best. I guess the main point is to motivate musicians to write more songs, to improve their skills and whatnot.

Anyway, my idea is to automate the process. I want to write some software that, given a title, can produce a half-plausible sounding song matching it. The first step would be to generate the lyrics, which I guess could be handled by an approach based on Markov chains trained on a range of existing song lyrics, probably combined with some sort of syllable and rhyming dictionaries to produce lines with some sort of rhythmic structure. Hmm, perhaps the rhythm should be generated first and then the lyrics made to fit it. The title would be included in one of the lines, probably near the beginning or end, or even several times in the chorus.

Next (or perhaps first), it would be necessary to generate some sort of plausible tune for the verse and chorus, and any other bits in-between (bridge? intro? instrumental bit in the middle?). I am not sure how to approach this, perhaps there is some literature on the subject. Maybe more Markov chains, or some sort of recombination of existing tunes, or some other method. I could really do with some advice from musicians here: how do you put together a tune that actually sounds tuneful, rather than just a list of random notes?

I assume that once the basic melody is generated, generating a passable accompaniment would be relatively straightforward, though again I will need to talk to some musicians.

Once the music and lyrics are sorted, along with an arrangement for the song, producing the final product would be a matter of running the lyrics and melody through Festival’s singing mode (as I have played with before), synthesise a few instruments and some drums for the rest of it, and mix it all together, ready to submit. Simple?

Anyway, reactions? Does this sound plausible? Has it all been done before? Can you offer any advice, especially for the musical side of it?

17 August 2009

An interesting tale of filehandles

Filed under: Computers — Tags: , , , , , , , , , , — qwandor @ 10:25 pm

I just found an interesting bug, so I thought it might be worthwhile to share it with these intertubes to prevent other people from making the same mistake.

I was just transferring some more music onto my iPhone with Amarok while at the same time listening to music. I wanted to listen to a particular track (from a Moby album I bought recently), so stopped and switched to that, but Amarok would not play it and complained about the sound device being busy. This seemed rather odd as it had just been playing fine until I tried to change tracks. Wanting to get to the bottom of this, I checked who had what open:

andrew@rata:~$ sudo lsof /dev/snd/*
lsof: WARNING: can't stat() fuse.sshfs file system /media/iphone
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
timidity 4162 root 6u CHR 116,1 4973 /dev/snd/seq
kmix 4511 andrew 10u CHR 116,0 5260 /dev/snd/controlC0
ssh 5605 andrew 16u CHR 116,0 5260 /dev/snd/controlC0
ssh 5605 andrew 33r CHR 116,33 4954 /dev/snd/timer
ssh 5605 andrew 39u CHR 116,16 5236 /dev/snd/pcmC0D0p
ssh 5605 andrew 41u CHR 116,0 5260 /dev/snd/controlC0
sshfs 5609 andrew 16u CHR 116,0 5260 /dev/snd/controlC0
sshfs 5609 andrew 33r CHR 116,33 4954 /dev/snd/timer
sshfs 5609 andrew 39u CHR 116,16 5236 /dev/snd/pcmC0D0p
sshfs 5609 andrew 41u CHR 116,0 5260 /dev/snd/controlC0

I should point out at this point that the way I get Amarok to sync music to my (jailbroken) iPhone is to FUSE-mount the iPhone via SFTP over the network, so that is why SSH was running. But on with the story.

SSH had my sound device open‽ What? That seemed very odd. I wondered whether perhaps there was some new SSH feature I had not heard about to forward sound over the network connection (as it can do for GUI applications using X11), but there I could find no mention of such a feature in the manpage or via Google, nor any possible reason why it might want to open a sound device. I asked lorne, and he was equally confused, but suggested a few things to check.

After a bit of poking around I discovered that I could not replicate this behaviour by mounting the filesystem myself, only when Amarok did it. This prompted a realisation of what must be happening: when Amarok launches sshfs to mount the iPhone filesystem, it presumably does a fork and exec to start the new process. But when you do this, the new process inherits all the open file handles of the parent process. Amarok of course had the sound device open to play the music I was listening to originally, so sshfs and subsequently ssh ended up with the same device file open. Amarok must then have closed it and tried to reopen it when I switched tracks, and this failed because the other processes it had launched still had it open. Of course.

So, lesson of the day: when you fork, remember to close any excess files before you exec. Especially if they are device files or other special files.

I should probably file a bug report for Amarok, but I am not sure that I can be bothered.

30 July 2009

Juvenile humour

Filed under: Computers, Humourous, Lists — Tags: , , , , — qwandor @ 9:32 pm

It turns out that if you take the title of a book and append the phrase “in your pants”, the resulting title is often more amusing than the original. Here I present you some examples:

About the Size of it: The Common Sense Approach to Measuring Things in Your Pants
The Adventures of Huckleberry Finn in Your Pants
All About Worship: Insights & Perspectives on Worship in Your Pants
Being Human in Your Pants
Calculus: A New Horizon in Your Pants
Computer Networking: A Top-down Approach Featuring the Internet in Your Pants
Computer Organization and Design: The Hardware/Software Interface in Your Pants
Coping with Controversy in Your Pants
The Diamond Age: Or, a Young Lady’s Illustrated Primer in Your Pants
Dick Smith’s Fun Way into Electronics in Your Pants
God’s Undertaker: Has Science Buried God in Your Pants?
The Snow in Your Pants

For some automated humour, I have just written a little script that will transform the titles of your books most recently added to LibraryThing in similar manner. Take a look at what it does for my books, or try it on your or your friend’s collection.

Oh, and on the topic of pants, if you have not read Star Wars pants quotes, perhaps you will be amused to do so. I cannot find the original site that I remember from years ago, but they are still funny. The premise is to take a line from Star Wars and replace one word with the word ‘pants’, thus improving the line.

Well, it is said that small things amuse small minds.

8 July 2009

Updates to theQuotebook

Filed under: Computers, thequotebook — Tags: , , , — qwandor @ 9:46 pm

I have made a few updates to theQuotebook over the past few weeks (well, okay, months). For those who have not already noticed:

  • You can now get an email when you are quoted, or when someone comments on one of your quotes (either a quote that you added or one that you said). This will be enabled by default the first time you log in. Users who have been created to be quoted but have not themselves logged in to the site will not get any emails. I originally implemented it so that they would get emails and so could see when they are quoted and log in if they wanted, but this was changed on request from a user. Let me know which way you think is best.
  • It is now a bit easier to sign up if you do not already have an OpenID, as the site now has an option on the login page to get one from myOpenID. The link there will sign you straight in once you have it, thanks to myOpenID’s affiliate programme.
  • It is possible to use some basic formatting in quotes and comments. You can make text *bold* and _italic_ like so, and also use BBCode if you are familiar with that. The formatting options for quotes are fairly limited, but comments allow most BBCode.
  • There is are now Atom feeds for comments as well as for quotes, so you can subscribe to both in your favourite feedreader (I recommend Google Reader) and keep up with them without checking the site all the time. To subscribe to the quotes and comments on your quotebooks, go to the home page, make sure you are logged in, and then click the RSS feed icon in your web browser (in Firefox it is a little orange icon in the address bar, right-aligned after the URL).

There have also been many minor improvements, such as the ability to add a new quote straight from the page for each quotebook, rather than having to click a link first.

Looking forward, we will hopefully have a new style for the site in the not-too-distant future (thanks to Daniel Sherson). If you are particularly curious, more plans and ideas may be found. Or, of course, you could talk to me.

Comments, suggestions, ideas and other feedback are of course welcome. I make no promises about doing what you say, but I will certainly be happy to listen to anything.

29 May 2009

Bob

Filed under: Computers, Humourous — Tags: , , , — qwandor @ 7:32 pm

I have been told that I should tell you all to talk to Bob. Bob is a chatterbot I wrote about three and a half years ago, which learns from what people say to it. Everything it says is based on something it has heard in the past; there is no knowledge of any particular language built into it.

Unfortunately the problem with putting such a thing on the web — or indeed most anywhere on the Internet — is that people tend to type garbage into it most of the time, and so it ends up spouting the same garbage. I have just now cleared its database though, so it should hopefully be relatively sensical for at least a few weeks until it gets overwhelmed with nonsense again. It can even be quite funny at times.

If you are curious, after chatting to it for a while you might want to see Bob chat to itself or see the contents of its ‘brain’.

Older Posts »

Create a free website or blog at WordPress.com.