This post describes the implementation in Python of a “Least Frequently Used” (LFU) algorithm cache eviction scheme with complexity O(1). The algorithm is described in this paper written by Prof. Ketan Shah, Anirban Mitra and Dhruv Matani. The naming in the implementation follows the naming in the paper. LFU cache eviction scheme is useful for... » read more
Using the Cambridge (Massachusetts) GIS data, we can compute some interesting geospatial statistics. Subway stations Using the address points (20844 points) and the subway stations data files, we can find out how many Cambridge addresses are in a certain radius of a Cambridge subway station. More particularly, we want to find the percentage of Cambridge... » read more
Using the Massachusetts Census 2010 Towns data source, we generated the following maps and statistics. The software used relies on technologies such as Python and PostGIS. Note: The data source has been generated and published by MassGIS. Population change We can see some towns in Cape Cod and Western Massachusetts not growing as fast (few... » read more
This post describes how Pytolab was designed to process Tweets related to the 2012 French presidential election, in real-time. This post also goes over some of the statistics computed over a period of 9 months. Note: I presented this project at EuroSciPy 2012: abstract. ArchitectureStatistics Architecture The posts are received from the Twitter streaming API... » read more
This post describes the implementation of sentiment analysis of tweets using Python and the natural language toolkit NLTK. The post also describes the internals of NLTK related to this implementation. Background The purpose of the implementation is to be able to automatically classify a tweet as a positive or negative tweet sentiment wise. The classifier... » read more
This post describes how dictionaries are implemented in the Python language. Dictionaries are indexed by keys and they can be seen as associative arrays. Let’s add 3 key/value pairs to a dictionary: The values can be accessed this way: The key ‘d’ does not exist so a KeyError exception is raised. Hash tables Python dictionaries... » read more
This article describes how string objects are managed by Python internally and how string search is done. PyStringObject structure New string object Sharing string objects String search PyStringObject structure A string object in Python is represented internally by the structure PyStringObject. “ob_shash” is the hash of the string if calculated. “ob_sval” contains the string of... » read more
This article describes how integer objects are managed by Python internally. An integer object in Python is represented internally by the structure PyIntObject. Its value is an attribute of type long. To avoid allocating a new integer object each time a new integer object is needed, Python allocates a block of free unused integer objects... » read more
We are going to talk about the toolkit pycrypto and how it can help us speed up development when cryptography is involved. Hash functions Encryption algorithms Public-key algorithms Hash functions A hash function takes a string and produces a fixed-length string based on the input. The output string is called the hash value. Ideal hash... » read more
This post describes the CPython implementation of the list object. CPython is the most used Python implementation. Lists in Python are powerful and it is interesting to see how they are implemented internally. Following is a simple Python script appending some integers to a list and printing them. As you can see, lists are iterable.... » read more