DreamPie

DreamPie, check it out it is as good as they put it. It is the best python shell ever and I'm intent on using it for all my local python tinkering.

But... if only it worked over ssh! These days I spend 99% of the time working remotely meaning I can't use this gem. Also, locally it still doesn't beat ipython's "sh" profile, so for the time being, I'll be using ipython as my main shell as usual but for local stuff I completely recommend it over anything including the incredible ipython and that is hard to accomplish.

Misc stuff and Programmers day.

Today is programmers day! In Russia. Anyway I have not been commenting for a long time so I'm trying to get into it again.

Do you guys remember when I said I spend to much time reading atheist blogs? Well I find now I find myself watching to much atheist channels in youtube, the upside is that I watch them more for the humor than actual debating. Debating creationists is seems more and more with each day.

In the same spirit it feels like I'm always feeding the trolls in slashdot because the posts that deserve to be replied the most are almost always made by irrational people.

One interesting post I wanted to share: Interpolation surprise

And finally let's celebrate the apology from the British government to Alan Turing! This is just great really.

ObjectMap

I like how Python ensures no library messes with my built-ins. The fact that neither I can monkey patch built-ins is however a pain, interestingly while built-ins can't be extended they can be decorated/overriden, and lo I present unto you, ObjectMap.

class ObjectMap(object):
  def __init__(self):
      """Did you look what I did there?"""
      self.map = map
  def __call__(self, *args, **kwargs):
      """The original map call"""
      return self.map(*args, **kwargs)
  def __getattr__(self, name):
      """map.name(sequence, args) == [item.name(args) for item in sequence]
      Here is the fun part"""
      def fn(seq, *args, **kwargs):
          return self.map(lambda item: getattr(item, name)(*args, **kwargs) , seq )
      return fn
map = ObjectMap():
words = "red green blue".split()
WORDS1 = map(lambda word: word.upper(), words)
WORDS2 = [word.upper() for word in words]
WORDS3 = map.upper(words)
assert WORDS1 == WORDS2 == WORDS3

Needless to say, this shouldn't be used in production code (so use it copiously).

IntegerDateTime

So I try once and again to get into the Python ORM wave, this time with the elegant Elixir library.

The way I see it, Elixir/SqlAlchemy or any of the other ORM libraries can make a lot for you, provided you do the right things from the beginning. Integrating it into my current work flow is just too much work and I'm in constant fear everything will crumble down at some point and I'll have to rewrite everything.

Anyway the problem I had, which I pasted into stackoverflow involved our consistent use of integer columns instead of datetime columns in mysql, if I wanted to make a table wrapper I needed to cover that case so in the end I wrote my own schalchem data type (also pasted at stackoverflow)

import datetime, time
from sqlalchemy.types import TypeDecorator, DateTime
class IntegerDateTime(TypeDecorator):
  """a type that decorates DateTime, converts to unix time on
  the way in and to datetime.datetime objects on the way out."""
  impl = DateTime
  def process_bind_param(self, value, engine):
      """Assumes a datetime.datetime"""
      assert isinstance(value, datetime.datetime)
      return int(time.mktime(value.timetuple()))
  def process_result_value(self, value, engine):
      return datetime.datetime.fromtimestamp(float(value))
  def copy(self):
      return IntegerDateTime(timezone=self.timezone)

In the end tough, it wasn't very useful because I don't have another table to which I can link this one so the main reason to write a wrapper for this class was void. Also, the query syntax was less nice than the SqlSoup auto-generated one so I should probably just use SqlSoup.

I still think Elixir/SqlAlchemy mappers are great, I understand they do a lot of stuff for you, like data definition centralization. But I just can't get a chance to use them where they aren't a hindrance!

So sad...

Batch Iterator and obscure Python details

I love Python generators and iterators, when they aren't making the easy trivial they are making the impossible possible.

I specially like to use iterators in streaming situations, like when reading from very large files or a database, because you don't have to traverse the sequence twice.

However in very large sequences I have had the need to perform some action every n items. I had the idea of using an special iterator that could split a sequence in sub-sequences but then I have to step over every item twice, once to pack it into the sub-sequence and once again to process it. Using islice was my first idea, but I needed to, somehow, comunicate to the "outer" iterator that the sequence has been exhausted or else I'd be stuck in an infinite loop iterating over empty subsequences.


I tough about adding an is_exhausted property to the sub, sequences, then I found out something interesting, you can't stuff properties into standard iterators, including those you get with generator expressions.

>>> i = iter([])
>>> i.is_exhausted = True
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    i.is_exhausted = True
AttributeError: 'listiterator' object has no attribute 'is_exhausted'
>>> def generator():
    yield True  
>>> g = generator()
>>> g.is_exhausted = True
Traceback (most recent call last):
  File "<pyshell#6>", line 1, in <module>
    g.is_exhausted = True
AttributeError: 'generator' object has no attribute 'is_exhausted'
>>>

No prob, I though, I can make everything inside a single generator! Actually I can't, once a generator raises StopIteration it can't do anything else.

>>> def anotherGenerator():
    yield 1
    yield 2
    raise StopIteration
    yield 3
    yield 4
>>> a = anotherGenerator()
>>> a.next()
1
>>> a.next()
2
>>> a.next()
Traceback (most recent call last):
  File "<pyshell#16>", line 1, in <module>
    a.next()
  File "<pyshell#12>", line 4, in anotherGenerator
    raise StopIteration
StopIteration
>>> a.next()
Traceback (most recent call last):
  File "<pyshell#17>", line 1, in <module>
    a.next()
StopIteration
>>> a.next()
Traceback (most recent call last):
  File "<pyshell#18>", line 1, in <module>
    a.next()
StopIteration
>>>

Ok, so I thought about using a custom class for the sub-sequences, one that stored a reference to the "parent" iterator, but then I thought, Why making two new classes if the parent is simply returning iterators why don't return self? This lead to the first working implementation:

class ibatch(object):
    """A batch iterator by rgz"""
    def __init__(self, sequence, size):
        """ibatch(iterable, size) -> sequence of iterables
        splits an iterable into groups of 'size' items lazily"""
        self.__sequence = iter(sequence)
        self.__size = size
        self.__counter = 0
    def __repr__(self):
        return "<batch iterator at %s>" % hex(id(self))
    def __iter__(self):
        return self
    def next(self):
        if self.__counter:
            if self.__counter > self.__size:
                self.__counter = 0
                raise StopIteration
            else:
                self.__counter += 1
                # When this raises StopIteration, it's the end.
                return self.__sequence.next()
        else:
            self.__counter = 1
            return self

This one uses two magic constants internally but is overall nice and compact, this is a demonstration of how it runs:

>>> for enum, items in enumerate(ibatch(xrange(10), 3)):
    print "Block #%s" % enum
    for item in items:
        print item,
    print '\n--'
  
Block #0
0 1 2
--
Block #1
3 4 5
--
Block #2
6 7 8
--
Block #3
9
--

Pretty nice, as long as the number of items isn't exactly divisible by the size of the sub-sequence, when that happen we get an empy sub-sequence complete with empty header and footer sections:

>>> for enum, items in enumerate(ibatch(xrange(9), 3)): # 9 items intead of 10...
    print "Block #%s" % enum
    for item in items:
        print item,
    print '\n--'

   
Block #0
0 1 2
--
Block #1
3 4 5
--
Block #2
6 7 8
--
Block #3

--

See that empty block? We can't get rid of it, because we don't know if the current sub-sequence is empty unless we try to get an item from it. This breaks a little of the conceptual cleanness of iterators, if the header depends on the sequence to not be opened first. However most of the time it is not a problem and it is very convenient, what we do is that we preload the first item in the sub-sequence to find out if the sequence is empty or not:

>>> class ibatch(object):
    """A batch iterator by rgz"""
    def __init__(self, sequence, size, preloading = False):
        """ibatch(iterable, size) -> sequence of iterables splits an iterable
        into groups of 'size' items lazily"""
        self.__sequence = iter(sequence)
        self.__size = size
        self.__counter = 0
        if preloading:
            self.next = self._next_preloading
        else:
            self.next = self._next
        assert self.next
    def __repr__(self):
        return "<batch iterator at %s>" % hex(id(self))
    def __iter__(self):
        return self
    def _next(self):
        if self.__counter:
            if self.__counter > self.__size:
                self.__counter = 0
                raise StopIteration
            else:
                self.__counter += 1
                # When this raises StopIteration, it's the end.
                return self.__sequence.next()
        else:
            self.__counter = 1
            return self
    def _next_preloading(self):
        if self.__counter == 0:
            self.__preloaded = self.__sequence.next()       
            self.__counter = 1
            return self
        elif self.__counter == 1:
            self.__counter = 2
            return self.__preloaded
        elif self.__counter <= self.__size:
            self.__counter += 1
            # When this raises StopIteration, it's the end.
            return self.__sequence.next()
        else:
            self.__counter = 0
            raise StopIteration           
 

So this class takes a preloading argument and choses the apropiate next method, I'm soo clever! Except it doesn't work.

>>> for enum, items in enumerate(ibatch(xrange(9), 3)):
    print "Block #%s" % enum
    for item in items:
        print item,
    print '\n--'

   
Traceback (most recent call last):
  File "<pyshell#43>", line 1, in <module>
    for enum, items in enumerate(ibatch(xrange(9), 3)):
TypeError: iter() returned non-iterator of type 'ibatch'
 

Wait what? how is it not an iterator? The minimal requisites for the iteration protocol are the __iter__ and next methods and it has both right? Unless iter() expects the class to have a next method, so I added it to the ibatch class:

def next(self):
    pass
 

This still doesn't work, but for an entirely different reason...

>>> for enum, items in enumerate(ibatch(xrange(9), 3)):
    print "Block #%s" % enum
    for item in items:
        print item,
    print '\n--'

   
Block #0
Traceback (most recent call last):
  File "<pyshell#48>", line 3, in <module>
    for item in items:
TypeError: 'NoneType' object is not iterable
 

What NoneType? It is talking about the return of the next method, so it is calling the next method in the class! Now it makes sense that iter() looks for it in the class definition, in other words the for statement doesn't call foo.next(), it calls foo.__class__.next(foo).

I understand why they don't want to use method resolution over and over in each iteration but grabbing a reference to the next method in the instance is the right thing to do, in my opinion. A dirty fix is calling the instance method in the class method like this:

def next(self):
    return self.next()
 

But that's innefficient, the most readable solution seems to be using two clases like this:

FINAL VERSION

class ibatch(object):
    """A batch iterator by rgz that doesn't creates empty batches"""
    def __init__(self, sequence, size):
        """ibatch(iterable, size) -> sequence of iterables
        splits an iterable into groups of 'size' items lazily"""
        self.__sequence = iter(sequence)
        self.__size = size
        self.__counter = 0
    def __iter__(self):
        return self
    def next(self):
        if self.__counter == 0:
            self.__preloaded = self.__sequence.next()       
            self.__counter = 1
            return self
        elif self.__counter == 1:
            self.__counter = 2
            return self.__preloaded
        elif self.__counter <= self.__size:
            self.__counter += 1
            # When this raises StopIteration, it's the end.
            return self.__sequence.next()
        else:
            self.__counter = 0
            raise StopIteration           

class ibatch_strict(object):
    """A batch iterator by rgz"""
    def __init__(self, sequence, size, preloading = False):
        """ibatch(iterable, size) -> sequence of iterables
        splits an iterable into groups of 'size' items lazily
        it is strict because it doesn't open the subsequence
        before the header is procesed but in turn it can leave
        an empty batch at the end if (len(sequence) % size) == 0"""
        self.__sequence = iter(sequence)
        self.__size = size
        self.__counter = 0
    def __iter__(self):
        return self
    def next(self):
        if self.__counter:
            if self.__counter > self.__size:
                self.__counter = 0
                raise StopIteration
            else:
                self.__counter += 1
                # When this raises StopIteration, it's the end.
                return self.__sequence.next()
        else:
            self.__counter = 1
            return self
 

As you can see I removed the __repr__ methods since they have nothing interesting to say, I also decided to make preloading the default class because I like it better ^^. Here is how it runs:

>>> for enum, items in enumerate(ibatch(xrange(9), 3)):
    print "Block: %s" % enum
    for item in items:
        print item,
    print "\n--"

Block: 0
0 1 2
--
Block: 1
3 4 5
--
Block: 2
6 7 8
--

So the lesson of today is: "python iterators use foo.__class__.next(foo) not foo.next()"
I'll try not forgetting that.

I hope this class is usefull for someone.

import time

Today is a special date, well actually an special UNIX date, according to the UNIX calendar -which is measured in seconds after January 1 1970- we are approaching date 1234567890 at precisely 18:31:30.

So in anticipation, let's write a little clock script:

import time
while True:
    print int(time.time())
    time.sleep(1)

Or if you prefer a countdown clock like myself:

import time
t = True
while t:
    t = 123456789 - int(time.time())
    print t
    time.sleep(1)
else:
    print "Happy valentine ^_^!"

Python vs Ruby on beautiful code, Red Beauty, Green Beauty

 Comparing Python vs Ruby is kind of a sport, this time I'll talk about code beauty. Python's huge advantages are its mature and insightful libraries and its faster run time. Grammar-wise, they are awfully similar.

 Ruby is basically an slower Python where there are no functions; methods can't be freely passed around and are called on reference without parens; lambdas can be defined in-line (blocks); monkey-patching runs wild and has a virtually endless stock of little conveniences and shortcuts.

 Its actually no small loot, the niceties cut the character count and the more obscure shortcuts you know the more compact you can make your code. You can see dramatic differences on code length between beginner and expert Ruby devs.

 This is what I call Red Beauty: Ruby focuses on making code easier to write.

 Some of the features are simple trade-offs and I feel Python makes the right choices more often, I prefer the slot based philosophy of object orientation and having to use parens on methods is an small price to pay.

 Functions vs blocks are a false dichotomy, multiline lambdas could solve both problems but if I have to choose I prefer first class function objects, you can't pass more than one block to a method in Ruby.

 The near ban on monkey-patching can be painful, mostly in your pride, since its cooler to use your own methods on strings than wrappers, but I'll argue that its thanks to this Python has better libraries. Python libraries will always be superior period, expect me to byte my tongue in six years, but I think the philosophies of Python make for better module writing.

 So what about the niceties and shortcuts? Well its a mixed bag... I miss string interpolation but that's about it. The different ways to turn an string into a hash actually bother me because it means I have to learn many ways to do something to understand somebody else's code and code written using spacial case shortcuts can need complete re-writing when the specs change.

 So this is what I call Green Beauty: Python focuses on making code easier to maintain.

 So which is more beautiful? Both, they just have a different shade of beauty.
 

PHP5 Iterators. MySQL iterator example.

PHP is stupid, enough said. Recently I wanted to abstract a table printing function so it could work with either arrays and mysql. In Python this is screams iterator and since I heard PHP5 supported iterators I alway wanted to write one. So before get to the PHP let me explain the Python way first:

The Pythonic Iterator Protocol:

1. Take the object to traverse, call its '__iter__()' method to obtain/initialize it
2. Call its 'next()' to obtain the current item
3. Exit from the iteration when 'next()' raises the 'StopIteration'

Simple isn't it? All the work is done in 'next()' and all it has to do is return a value or raise 'StopIteration'


The PHP Iterator Protocol:

1. Call 'rewind()' to make sure we are iterating from the begining.
2. Call 'valid()', if it returns false exit from the iteration.
3. Take the first element by calling 'current()' fetching the first element.
4. Optionally get the key of the first element by calling 'key()'.
5. Call 'next'()' to do whatever is necesary to fetch next item, ignore the return value.
6. Call 'valid()', if it returns false exit from the iteration.
7. Take the next element by calling 'current()'.
8. Optionally get the key of the next element by calling 'key()'.
9. Call 'next'()' to do whatever is necesary to fetch next item, ignore the return value.
10. Repeat steps from 6 to 9.

"Wait a minute!" you say "steps 2-5 are the same that steps 6-9!" No they aren't. Steps 6-9 operate in the "next" item, the one 'next()' fetched for us. steps 2-5 operate on some ghostly "first" item that nobody has fetched yet.

So 'valid()', 'current()' and 'key()' have to behave differently for the first run. In practice it's sufficient with calling 'next()' from within 'valid()' the first time. But the two resons why this is horrible are because...

OOP and semantic purity are like M. Night Shyamalan and plot twists:

One implies the other, and it hurts when it doesn't match our expectatives. In OOP methods are named in a way that you know what they do just from looking at its name. The boolean method 'valid()' suggest a simple procedure to ensure the currently selected item is part of the iteration you don't expect it to also fetch the first item. Another problem is one of efficiency, for an array with N elements 'valid()' will have to make a test N times where it will evaluate the same allways except the very first case.

No, we have to take the inicialization out of the loop. OOP principles tell us the constructor is the place to make these set ups. But there is a problem, 'rewind()' is called just before the iteration begins! So we find ourselves in a dichcotomy:

1. Fetch the first item in '__construct()', make 'rewind()' do nothing.
2. Fetch the first item in 'rewind()', that is, call 'next()' after rewinding.

Either way 'rewind()' is a lier because it doesn't do what its name says it does. Now if I have to choose the leser evil, option 2 is the way to go, because it makes the iterator reusable which is the purpose of calling 'rewind()' in the first place. And so hereby I present:

A simple PHP MySQL Iterator:

class mysqlIter implements Iterator{
    private $resource;

    private $count = 0;
    private $pos = -1;
    private $valid;
    private $curval;
    public function __construct($resource){
        $this->resource = $resource;
    }
    public function next(){
        if ($value = mysql_fetch_assoc($this->resource)){
            $this->valid = true;
            $this->curval = $value;
            $this->pos++;
        } else {
            $this->valid = false;
        }
    }
    public function valid(){
        return $this->valid;
    }
    public function current(){
        return $this->curval;
    }
    public function key(){
        return $this->pos;
    }
    public function rewind(){
        mysql_data_seek($this->resource, 0);
        $this->next();
    }
    public function count(){
        return mysql_num_rows($this->resource);
    }
}

Aftermat.

At first I wasn't aware 'next()' was not going to get called until the second leap, then 'rewind()' started to mess up the result, so it took me a little longer to implement the iterator. I blame the PHP way and its documentation.

A php-head will tell me that this is a case of PHP just being a different language, not stupid but the devil is in the details. For instance it is a good argument to say that there is nothing incosistent on rewind calling next() because it means a manually rewinded iterator is pointing to its first item always but this opens the question, why would you manually access the first item in an iterator? The answer is because you aren't exactly handling an iterator but a data structure that is iterable. Iteration happens direclty to the object, in Pythonland most iterables actually use a proxy iterator object (that's the purpose of '__iter__()') which means, among other things, that iterable objects don't need to contain iterator related attributes o methods.

Iterable objects in Python don't usually carry an internal pointer or implement next(), they simply have an '__iter__()' method that returns an object that does so.

Another implication is that, because a new iterator is instantiated on demand every time, the same data structure can be traversed by multiple clients without conflicts unlike PHP iterators.

But there are other problems with the argument that 'rewind()' calling 'next()' ensures the internal pointer is at the right position. One of them is that, if directly accessing an iterable is so desirable, then one would expect people to access freshly instantiated iterators. That means '__construct()' also should call 'next()', just in case.

But if an iterator is instantiated and then used (a very common pattern) then the first item would have been fetched twice!!

In short, iterators in PHP5 suck.