The problem was indeed that I was running it on a 32 bit machine, where the hash function return a negative number for the char ‘z’, that’s the reason, why perturb doesn’t reach 0 and every table slot is probed.

And the hash function returns a different number, because the char pointer size is 4 bytes on 32bit and 8 bytes on 64bit.

thanks for this great article, it really helps understanding Python’s dict implementation.

I tried to implement the probing algorithm in python:

https://gist.github.com/3724747

As you can see I’m trying to insert several ‘z’ chars into the table, which results in multiple collisions.
What I don’t understand is, after the 5th ‘z’ gets inserted, why doesn’t the probing algorithm ever terminate and never reach one of the empty table slots 1, 2 or 4?
When perturb reaches -1 (I’m on a 32bit OS btw) the probing slot just toggles between slot 3 and 7.

Can this somehow be the correct behaviour, because more than 5 consecutive collisions is very unlikely?
Do you see any mistakes?

BTW @David
Your calculated sequence is the sequence for a table with 32 slots, which makes the mask 31 and the first slot lookup hash(‘z’) & 31 = 27
Whereas Laurent used the 3 for the initial slot, which came from hash(‘z’) & 7 example, but then used 31 for the mask from the 32 slot table example.

i = 3
mask = 31
perturb = hash(‘z’)
while True:
i = (i < < 2) + i + perturb + 1
slot = i & mask
print slot
perturb >>= 5

“Just out of curiosity, let’s look at the probing sequence when the table size is 32 e.g. mask = 31
3 -> 11 -> 19 -> 29 -> 5 -> 6 -> 16 -> 31 -> 28 -> 13 -> 2…”

By my calculation, the sequence is:
3 11 21 29 30 8 11 24 25 30 23 20 5 26 3 16 17 22 15 12 29 18 27 8 9 14 7 4 …

Can you please confirm your results?
Thanks!

The following algorithm is from function lookdict_string(PyDictObject *mp, PyObject *key, long hash);
1 i is the current slot index
2 set perturb to hash
3 forever loop:
4 set i to i << 2 + i + perturb + 1
5 set slot index to i & mask
6 if slot is free:
7 return it
8 right shift perturb by 5 bits

I want to implement this algorithm in my own program to store a large number (about 13 M) of key/value pairs. (The keys are strings.) But I find that this algorithm fails to find a free slot after inserting 70,000 key/value pairs, though I set the hash table size to 30 M.

So my question is, does python support a dictionary at this scale (10M entries)?

Thanks!