Python multiprocessing adventures

Trying to figure out some multiprocessing code from last night.
As one would expect, the single thread code maxes out a single core and the other three(it’s a dual core HT chip) are very lightly loaded with random background crap.
For multiprocessing, the average load per core is in between the peak vs low cores for the single, and it close to the same for all four. Basically what you’d expect to see with smaller tasks spread around.
RAM usage remains flat, or too close to flat for Ubuntus System Monitor to notice a difference.
But, the runtime is about .8s longer. Less stress on CPU cores and same stress on RAM and it takes longer.
Hmm. Some work to do on optimizing the algorithm.

Bigfoot search

I have the lottery project for programming practice, but that would require much more statistics knowledge than I have to come up with the algorithm and to interpret the results.

Another I have might not require as much stats knowledge. Possibly some, because it’s looking for correlations, but not nearly as heavy on the numerical analysis. A bigfoot search.

Bigfoot is, of course, a legend of a large hairy primate running around North America. The big problem with that- there’s really no way a large primate could maintain a breeding population large enough to account for the legend, but small enough to escape scientific notice for so long.

But- while it’s rare, humans do occasionally get large and hairy enough to be mistaken for a bigfoot(and even normal sizes might be so identified under some viewing conditions). Humans also occasionally just disappear never to be seen again. It seems likely to me that if any Bigfoot sightings are actually of a large primate, Homo Sapiens is by far the most likely source.

So, my thought is to comb through missing persons data, and bigfoot sightings, and see if there are any good correlations between the sightings and the last known location of the missing person. Focusing mostly on any missing persons with gigantism and/or hypertrichosis. I won’t reject non-mutants from the analysis entirely, if someone

Is the Bigfoot legend based on humans with various mutations that left or were ejected from society? A handful of mutant humans could escape capture or positive ID in the wild indefinitely, and they don’t require a large breeding population that has somehow escaped notice.

Again, like the lottery idea, I’m not expecting much. But it would be useful practice in writing code to do data analysis, and has the added advantage of requiring less additional knowledge to implement it, so I can focus mostly on the programming itself.

Daily Programmer 239 Easy – Game of Threes

So, the latest Daily Programmer project I’m working on(I’ll get to finishing the last one, eventually) is this weeks Easy level project, Game of Threes.

This one wasn’t complicated, here’s the code.

def get_diff_from_multiple(n):
    """ Returns -1, 0, or 1, whichever will add to n to           
    make the sum divisible by 3
    if n % 3 == 2:
        return 1
    elif n % 3 == 1:
        return -1
    elif n % 3 == 0:
        return 0

The core of the program is to either divide by three, or tweak the number to make it divisible by three.  The easiest way I saw was to just check how far it was from a multiple of three, and add the appropriate value.  It was pretty clear that modulus 3 would be the easy way- a remainder of 2, means adding 1 will make the number divisible by 3.  A remainder of 1, just subtract 1.  Remainder of 0, no change needed.

The function to play the game runs a loop until n reaches 1, outputting the appropriate status on each iteration of the loop.

def game_of_threes(n):
    """ Plays the game of threes with starting number n """

    while n > 1:
        diff = get_diff_from_multiple(n)
        print("{} {}".format(n, diff))
        n = (n + diff) // 3


There you go. Simple design for a simple project, though I’m sure there are some cleverer ways to do it.

Daily Programmer 238 Intermediate: Fallout Hacking Game(Wordlist data structure design)

One of my current projects is the Daily Programmer challenge here.  It’s a project to write a program to play a version of a hacking minigame from the Fallout series.  Here, I’ll work through my design process for a Python 3.4 implementation.

I started by just throwing out a little bit of code.  I didn’t expect this to get me anywhere, at least nowhere good.  But sometimes, just typing a few things out can help you realize what a project requires that purely thinking through it doesn’t.

I quickly realized I needed a data structure of some sort to hold the list of potential words.  A simple list of words would work, but depending on my source wordlist, I might end up with many words longer and shorter than the 4-15 letters the game design specifies.  I’d also have to look through the entire list to retrieve the small handful of words that are part of a run of the game.  Simply using a list would mean I’d have to account for these issues in many places in the game code, and that’s a problem.

I could use a list, and then have functions I call to work with the list.  But then I’d have to worry about global variables and/or passing a list all over the place.  That can be quite a mess.  Not as bad as repeating my data access code, but still a mess.  So I decided to implement it as a class, to package the data and its access code together.

I decided first it needed a way to give the user a list of words of appropriate length.  From here, I tried to think of an appropriate internal representation, and the easiest seemed to be a list of lists, indexed by word length. With that, I can use random.sample() to easily get an appropriate length list.

I can also use a function to fill the internal data structure that rejects words with lengths outside the specified limits.  There’s no sense having words with length of 16 in the data structure when I never need words that long.

The complete class is as follows:

class HackingGameWordList: 
    def __init__(self, wordlist):
    """ Wordlist is a list of strings, constants determine
    what length words I want to pull from the list
        self._MIN_LENGTH = 4
        self._MAX_LENGTH = 15
        self.words = [[] for _ in range(self._MAX_LENGTH)]
    def fill_list(self, wordlist):
        """ Breaks up the full wordlist into the internal list of lists
        indexed by word length
        for word in wordlist:
            l = len(word)
            if l >= self._MIN_LENGTH and l <= self._MAX_LENGTH:
     def get_n_words(self, n, l):
         """ Return a list of n randomly selected words of length l """
         return random.sample(self.words[l], n)

NSA dreams of smartphones with “split” crypto keys protecting user data | Ars Technica

NSA dreams of smartphones with “split” crypto keys protecting user data | Ars Technica.

Ideally, the legal and technical difficulties involved in getting past someones encryption, and getting past the lock on their safe, should be the same.  After all, these two things do essentially the same job.  Some details of various laws and regulations might have to differ, but it should all aim towards the goal of the peoples rights being the same, and the governments authority being the same, and the ability of both parties to act on their interests being the same.

Failing perfect equivalence, I’d like to see policy err on the side of protecting privacy.  Artificial weaknesses introduced to force equivalence can spiral out of control.  Even if you trust the authorities won’t misuse it ever, if there’s a backdoor, other people can use it too if they find it.  And if you trust the authorities that much, you’re deluded.

Hopefully some of these proposals get us to a better balance of the need to stop bad guys, and the need to maintain privacy.  At least the NSA is indicating some willingness to compromise.  Probably not enough of a willingness, but it’s better than we’ve had so far.

OS X Yosemite install stupid

So, the OS X Yosemite installer checks your hard drive before installing.  Makes sense, right?  You don’t want something wrong with the drive leaving you with a broken installation.  Now, you’d think this would be the first thing the installer does- while most problems can’t be fixed while the drive is mounted, they can be checked for and the installer would refuse to proceed until the errors are fixed.

Except… the Yosemite installer does not do this check until after the point of no return.  If your drive fails the check, as mine did, the installation will fail and your original OS will be inaccessible.  I had no opportunity to even try to repair the disk to allow the upgrade to proceed.

Now, thankfully, at the point in the installation where this check is done, the Yosemite installer is booting into a minimal OS X environment.  Terminal and Disk Utility were both there, and the downloaded installer for the upgrade was still present.  This did allow me to create a bootable USB installer to attempt to repair the disk.  Unfortunately, the problems with the disk were too severe to be repaired.  It reformatted fine, so it appears to have just been some data corruption rather than a failing hard drive, still, it would have been nice to find this at a stage where I could shove all my important data out over the network as I didn’t have the money at the time to run out and get blank DVDs, as I’ve just moved and moving is expensive as fuck(at least when it’s your first apartment you are moving to).

Yosemite isn’t bad now that I’ve got it installed.  I’m not sure how much is due to improvements in Yosemite over Snow Leopard, or the elimination of accumulated cruft from SL or the Leopard install that was installed over, but this thing feels a good bit snappier.  I just wish Apple had made the sensible decision to run the disk verification *first*, leaving me with a full OS X install to bring to bear on the problem, or at least a functioning system I could use and only run the upgrade again once I had the media or network connectivity to easily back up some data.

NCIS and Computers

Apparently, you can write a virus that can go through the power cord on a laptop- not a power line networking device, but the actual power cord- to infect other systems.  This was actually how it was said to have accessed the NCIS network.

To give them more credit than they deserve, the operating state of an electrical device can generate some current fluctuation in its power cord, and in severe cases, even in other devices on the same circuit. If the circuit is anywhere near competently designed and constructed, this will be a very small fluctuation, but to be fair it would exist.

But to exploit it to the effects seen in tonights episode, the only way this is even in the same multiverse as plausible would require specific targeting of the network being attacked and the vector being used. You’d need to know exactly how your vector performs under varying operating conditions to reliably generate a signal, you’d need to know exactly how the circuit it’s attached to is designed and implemented, you’d need to know how other devices attached to it will affect things- someone plugging in a coffee machine you didn’t expect could throw the whole thing off by changing how the circuit reacts to what you do to it.  An electrician using an extra inch of wire could throw it all off. Microamps of current draw could throw it off.

I really wouldn’t be surprised if this would require such precise control of the currents you induce in the power cord that quantum mechanics makes it more random luck than the virus actually doing anything, unless the virus writer is waiting to hear back on his Nobel Prize in EVERYTHING FOREVER for coming up with a reliable means to predict quantum effects with absolute precision.

This isn’t even getting to the issues of how to jump from one circuit to another, or how the outbreak was restricted to just the NCIS network when it somehow could propogate over power lines without actual power line networking fast enough to take out their entire network in a second or two.