Are you are looking for an easy way to download your Twilio resources (calls, sms, notifications, recordings…) to your own database so you can access them faster and also when you are offline? Twilio Resources DB is a Python module doing just that.

Features

  • Download Twilio resources to a database.
  • Download recordings audio files.
  • Support for MySQL, PostgreSQL and Redis.
  • Automatic database tables creation for SQL DB.
  • Handle tables relations for SQL DB.
  • Option to download new resources continuously.

This is the list of resource types we support:

  • Accounts
  • Calls
  • Notifications
  • Recordings
  • Transcriptions
  • Sms Messages
  • Conferences
  • Incoming Phone Numbers
  • Outgoing Caller Ids

Requirements

We need to install simplejson so we can use the JSON format for the HTTP requests results: ie: resources returned using JSON instead of XML.

sudo easy_install simplejson

Depending on the database used, we need to install SQLAlchemy for MySQL/PostgreSQL or redis for Redis.

easy_install SQLAlchemy
easy_install redis

We also need the Twilio Python library helper to help us with the HTTP requests to the Twilio server:

git clone https://github.com/twilio/twilio-python.git
python setup.py install

At last, we need to install the Twilio resources DB library:

git clone https://github.com/laurentluce/twilio-python-utils.git
python setup.py install

Download resources to a database

Here is an example of a simple Python script using the library to download all the resources from the Twilio server to a MySQL database.

A “Resources” object needs to be created with a list of settings:

from twilioresourcesdb import resources

# settings
settings = {}
# Twilio account
settings['account_sid'] = 'Twilio account SID'
settings['account_token'] = 'Twilio account token'
# DB
settings['database_type'] = 'mysql'
settings['database_user'] = 'username'
settings['database_password'] = 'password'
settings['database_host'] = 'hostname or ip address'
settings['database_name'] = 'database name'
# Resources options
settings['check_frequency'] = 300 # check for new resources every 5 minutes
# Download recordings audio files
settings['download_recordings'] = True
settings['recording_path'] = '/data/recordings/'
settings['recording_format'] = 'wav'

# Instantiate resources object.
# This will setup a connection with the DB and create the table if necessary.
r = resources.Resources(settings)

Here is the full list of options:

  • account_sid : Twilio account SID – ‘ACxxxxx’
  • account_token : Twilio account token – ‘xxxxx’
  • database_type : type of database – ‘mysql’, ‘postgresql’, ‘redis’
  • database_user : database username – ‘xxx’ (default: ‘root’, not required for Redis)
  • database_password : database user password – ‘xxxx’ (default: None)
  • database_host : database ip address or hostname – ‘xxxx’ (default: ‘localhost’)
  • database_port : database port number – xxxx (default: 3306 for MySQL, 5432 for PostgreSQL, 6379 for Redis)
  • database_name : database name – ‘xxxx’ (not required for Redis database)
  • check_frequency : frequency in seconds on how often to check for new resources – xx (default: 5)
  • page_size : number of resources to download at each request – xxxx (default: 50)
  • download_recordings : enable recordings downloads (audio files) – True/False (default: False)
    • recording_path : absolute path to store recordings audio files – ‘/xxx/xxx/xxx…’
    • recording_format : audio files format: ‘wav’, ‘mp3’

Next, we can download all the resources and then stop or we can start a thread downloading the resources continuously.

First option – blocking call – download all resources then stop:

r.process()

Second option – start a thread to download resources continuously:

r.start()

Here is some code you may want to re-use to stop the thread when the script is interrupted:

while True:
  try:
    time.sleep(1)
  except (KeyboardInterrupt, SystemExit):
    r.stop = True
    r.join()
    break

When an in-progress resource is received from the server, it is placed in a temporary list and checked against for completion regularly so we only save completed resources in the database. For example, a queued SMS message will be placed in this temporary list and only saved to the database when it is in a termination state: ie: completed, failed, dropped…

MySQL and PostgreSQL database tables

When you create the Resources object, the tables are automatically created for you (not if they already exist).

Here is the list of tables created automatically:

  • accounts
  • calls
  • notifications
  • recordings
  • transcriptions
  • sms_messages
  • conferences
  • incoming_phone_numbers
  • outgoing_caller_ids

And those are the relationships using foreign keys:

  • calls -> accounts
  • sms_messages -> accounts
  • conferences -> accounts
  • incoming_phone_numbers -> accounts
  • outgoing_caller_ids -> accounts
  • notifications -> accounts
  • notifications -> calls
  • recordings -> accounts
  • recordings -> calls
  • transcriptions -> accounts
  • transcriptions -> recordings

Redis keys and values

In case of Redis, we add the resources using keys and values.

The resource’s key is the resource SID and the value is the JSON resource object.

Call resource: CAxxxx
SMS message resource: SMxxxx
…

That’s it for now. Don’t hesitate to add comments if you have any feedback.

Last modified: May 30, 2020

Author