Most of the tasks in data science are long-running, and many folks (me included) execute those tasks on remote machines. And the crucial thing for those tasks is logging: you do need to know how training process was going and see the learning curves. It would also be convenient if you could access those logs from anywhere and be notified when the process had finished. So I built the
cloudlog is a very simple Python logger that duplicates your console logs to a local file, saves a copy safely in the cloud, and can as well notify you via messenger bot. And it can do all those things with
pyplot plots as well! For cloud service I went with Dropbox, as it’s easy to integrate and can be accessed from any device such as your phone. For messenger I chose Telegram, being a huge fan of the platform.
How to use
- Install package:
pip install cloudlog
from cloudlog import CloudLog
- Log text by simply calling a
log = CloudLog(root_path='~/logs')) log('Some important stuff happening.') log('And again!') log('Luckily, it\'s all safe now in a local file.')
pyplotplots as images in the same folder:
from matplotlib import pyplot # Draw a plot x = range(42) pyplot.plot(x, x) pyplot.xlabel('Amount of logs') pyplot.ylabel('Coolness of your app') pyplot.grid(True) # Call it before calling `pyplot.show()`. log.add_plot() pyplot.show()
In order to sync your logs and plots to Dropbox, do the following.
- Create a Dropbox app with
App folderaccess type.
- Get your Dropbox access token and provide it in initialiser.
sync()in order to dispatch log file to your Dropbox app folder.
log = CloudLog(root_path='~/logs', dropbox_token='YOUR_DROPBOX_TOKEN_HERE') log('Some important stuff happening again.') log('Luckily, it\'s all safe now. In the cloud!') log.sync()
Plots are being synced to Dropbox folder by default.
You may as well get notifications in a Telegram chat, with logs and plots being sent to you.
- Create a Telegram bot.
- Get your Telegram Bot API access token
- Find out your Telegram chat or user ID.
- Provide both values in the initialiser.
log = CloudLog(root_path='~/logs', telegram_token='YOUR_TELEGRAM_TOKEN', telegram_chat_id='CHAT_ID') log('Some important stuff once more.') log('Luckily, it\'s all safe now in a local file. AND you\'re notified — how cool is that?') log.sync(notify=True, message='I\'m pregnant.')
Specify the same
notify flag for plots for them to be sent to a Telegram chat as well:
Since one may be tempted to dispatch a bunch of updates at the same time, the user will not be notified about messages containing files, such as plots and logs — only about the
message passed to
There you go! Your remote machine will now not only safely store your logs in the cloud, providing easy access from anywhere, but will as well send you notification with a full report.
You could have guessed that Fenton is the name of my remote machine, of course!