A collection of small Python scripts and tips.

This post is based on a Twitter thread I started in April 2020 and works as a centralized way to read all the tips in an easier format than Twitter's 280 characters.

Both will be updated frequently.

1
2
3
4
5
6

import itertools

test = [[-1, -2], [30, 40], [25, 35]]
list(itertools.chain.from_iterable(test))

>>  [-1, -2, 30, 40, 25, 35]

1
2
3
4

test =  [[-1, -2], [30, 40], [25, 35]]
map(int, ''.join(c for c in test.__str__() if c not in '[]').split(',') )

>> [-1, -2, 30, 40, 25, 35]

1
2
3
4
5
6
7
8

from collections import Counter
count = Counter(['a', 'b', 'c', 'a', 'a', 'b', 'd'])

print(count)
>> Counter({'a': 3, 'b': 2, 'c': 1, 'd': 1})

count['a']
>> 3

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

import itertools as it

data = it.cycle([1, 2])

for i in range(10):
    print(next(data))

>> 1
2
1
2
1
2
1
2
1
2

1
2
3
4
5
6
7
8

# slice(start, end, step)

STEPTWO = slice(None, None, 2)
integer_list = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

integer_list[STEPTWO]

>> [0, 2, 4, 6, 8]

1

integer_list[::2]

1
2
3
4
5
6

# slice(None, None, -1) or [::-1]

integer_list = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
integer_list[::-1]

>> [9, 8, 7, 6, 5 …

The Data

I decided to monitor all geolocalized tweets with my home city (Caracas) as origin over a period of five days, two days before the elections, the election day, and two days after it.

To fetch the data, I used the Twython library since I've worked with it before. Therefore, it should be easy to setup a quick script to fetch the data and save it in a text file.

Once the API and OAUTH keys are correctly following Twython's documentation, the next step is to fetch the data originating in certain location, passing it as a parameter to the streaming API. Twitter uses a set of bounding boxes to track the location of a tweet and only geolocated Tweets falling within the requested bounding boxes will be included.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

import json, os
from twython import TwythonStreamer

caracas_location = "-67.017225,10.419554,-66.778716,10.52006"

class MyStreamer(TwythonStreamer):
    def on_success(self, data):
        if data['coordinates'] is not None:
            with open('tweets.txt', 'a') as f:
                f.write(json.dumps(data))
                f.write('\n')

    def on_error(self, status_code, data):
        with open('errors.txt', 'a') as f:
            f.write('error: {0}: {1}'.format(status_code, data))



stream = MyStreamer(os.environ['APP_KEY'], os.environ['APP_SECRET'],
                        os.environ['OAUTH_TOKEN'], os.environ['OAUTH_TOKEN_SECRET'])

stream.statuses.filter(locations=caracas_location)

Since I just wanted to fetch geolocalized tweets, I checked for the existance of the coordinates entity in the …

Jpeg support in PIL and pillow.

1
2
3
4

$ sudo apt-get install libjpeg libjpeg-dev libfreetype6 libfreetype6-dev zlib1g-dev
$ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libfreetype.so /usr/lib/
$ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libjpeg.so /usr/lib/
$ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libz.so /usr/lib/

Installing lxml in Python (Debian based).

If you're getting the "fatal error: libxml/xmlversion.h: No such file or directory" error, just install the following development files:

1

$ sudo apt-get install python-dev libxml2-dev libxslt1-dev

Problems with 'matplotlib' and freetype.

If you're having problems installing matplotlib in a Python virtualenv, and are getting the 'freetype missing' error, you sould install the development files for freetype, and (in most cases) rebuild the python dependencies for matplotlib.

1
2

$ sudo apt-get  -u install libfreetype6-dev
$ sudo apt-get build-dep python-matplotlib

After that, you can just use pip normally to install matplotlib

1

$ pip install matplotlib

PS. I know of cases where you have to update python-virtualenv and python-pip after you use build-dep. Just apt-get upgrade your installation. Today I learned about the 'pydoc' command from Python.

Problems with locale

In some machines I've found problems when setting locales from Python. First check the results of running

1

$ locale

In my case is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

LANG=es_VE.UTF-8
LANGUAGE=es_VE:es
LC_CTYPE="es_VE.UTF-8"
LC_NUMERIC="es_VE.UTF-8"
LC_TIME="es_VE.UTF-8"
LC_COLLATE="es_VE.UTF-8"
LC_MONETARY="es_VE.UTF-8"
LC_MESSAGES="es_VE.UTF-8"
LC_PAPER="es_VE.UTF-8"
LC_NAME="es_VE.UTF-8"
LC_ADDRESS="es_VE.UTF-8"
LC_TELEPHONE="es_VE.UTF-8"
LC_MEASUREMENT="es_VE.UTF-8"
LC_IDENTIFICATION="es_VE.UTF-8"
LC_ALL=

Then use:

1
2
3
4
5
6

import locale

try:
    locale.setlocale(locale …