Cloud logger

Most of the tasks in data science are long-running, and many folks (me included) execute those tasks on remote machines. And the crucial thing for those tasks is logging: you do need to know how training process was going and see the learning curves. It would also be convenient if you could access those logs from anywhere and be notified when the process had finished. So I built the cloudlog!

cloudlog

cloudlog is a very simple Python logger that duplicates your console logs to a local file, saves a copy safely in the cloud, and can as well notify you via messenger bot. And it can do all those things with pyplot plots as well! For cloud service I went with Dropbox, as it’s easy to integrate and can be accessed from any device such as your phone. For messenger I chose Telegram, being a huge fan of the platform.

How to use

• Install package:

pip install cloudlog

• Import CloudLog class:

from cloudlog import CloudLog

• Log text by simply calling a CloudLog instance:

log = CloudLog(root_path='~/logs'))
log('Some important stuff happening.')
log('And again!')
log('Luckily, it\'s all safe now in a local file.')

• Add pyplot plots as images in the same folder:

from matplotlib import pyplot

# Draw a plot
x = range(42)
pyplot.plot(x, x)
pyplot.xlabel('Amount of logs')
pyplot.ylabel('Coolness of your app')
pyplot.grid(True)

# Call it before calling `pyplot.show()`.
log.add_plot()

pyplot.show()

Dropbox

In order to sync your logs and plots to Dropbox, do the following.

• Create a Dropbox app with App folder access type.
• Get your Dropbox access token and provide it in initialiser.
• Call sync() in order to dispatch log file to your Dropbox app folder.

log = CloudLog(root_path='~/logs', dropbox_token='YOUR_DROPBOX_TOKEN_HERE')

log('Some important stuff happening again.')
log('Luckily, it\'s all safe now. In the cloud!')
log.sync()

Plots are being synced to Dropbox folder by default.

Telegram

You may as well get notifications in a Telegram chat, with logs and plots being sent to you.

• Create a Telegram bot.
• Get your Telegram Bot API access token
• Find out your Telegram chat or user ID.
• Provide both values in the initialiser.

log = CloudLog(root_path='~/logs', telegram_token='YOUR_TELEGRAM_TOKEN', telegram_chat_id='CHAT_ID')

log('Some important stuff once more.')
log('Luckily, it\'s all safe now in a local file. AND you\'re notified — how cool is that?')

log.sync(notify=True, message='I\'m pregnant.')

Specify the same notify flag for plots for them to be sent to a Telegram chat as well:

...
log.add_plot(notify=True)

Since one may be tempted to dispatch a bunch of updates at the same time, the user will not be notified about messages containing files, such as plots and logs — only about the message passed to sync() method.

There you go! Your remote machine will now not only safely store your logs in the cloud, providing easy access from anywhere, but will as well send you notification with a full report.

You could have guessed that Fenton is the name of my remote machine, of course!

Follow @alexstaravoitau Star Fork Download