Automating ML&systems R&D

After releasing CK we started working with the community to gradually automate the most common and repetitive tasks for ML&systems R&D (see the journal article and FastPath’20 presentation).

We started adding the following CK modules and actions with a unified API and I/O.

Platform and environment detection

These CK modules automate and unify the detection of different properties of user platforms and environments.

  • module:os [API] [components]
  • module:platform [API]
  • module:platform.os [API]
  • module:platform.cpu [API]
  • module:platform.gpu [API]
  • module:platform.gpgpu [API]
  • module:platform.nn [API]

Examples:

ck pull repo:octoml@mlops

ck detect platform
ck detect platform.gpgpu --cuda

Software detection

This CK module automates the detection of a given software or files (datasets, models, libraries, compilers, frameworks, tools, scripts) on a given platform using CK names, UIDs, and tags:

It helps to understand a user platform and environment to prepare portable workflows.

Examples:

ck detect soft:compiler.python
ck detect soft --tags=compiler,python
ck detect soft:compiler.llvm
ck detect soft:compiler.llvm --target_os=android23-arm64

Virtual environment

  • module:env [API]

Whenever a given software or files are found using software detection plugins, CK creates a new “env” component in the local CK repository with an env.sh (Linux/MacOS) or env.bat (Windows).

This environment file contains multiple environment variables with unique names usually starting from CK_ with automatically detected information about a given soft such as versions and paths to sources, binaries, include files, libraries, etc.

This allows you to detect and use multiple versions of different software that can easily co-exist on your system in parallel.

Examples:

ck detect soft:compiler.python
ck detect soft --tags=compiler,python
ck detect soft:compiler.llvm

ck show env 
ck show env --tags=compiler
ck show env --tags=compiler,llvm
ck show env --tags=compiler,llvm --target_os=android23-arm64

ck virtual env --tags=compiler,python

Meta packages

When a given software is not detected on our system, we usually want to install related packages with different versions.

That’s why we have developed the following CK module that can automate installation of missing packages (models, datasets, tools, frameworks, compilers, etc):

This is a meta package manager that provides a unified API to automatically download, build, and install packages for a given target (including mobile and edge devices) using existing building tools and package managers.

All above modules can now support portable workflows that can automatically adapt to a given environment based on soft dependencies.

Examples:

ck install package --tags=lib,tflite,v2.1.1
ck install package --tags=tensorflowmodel,tflite,edgetpu

See an example of variations to customize a given package: lib-tflite.

Scripts

We also provided an abstraction for ad-hoc scripts:

See an example of the CK component with a script used for MLPerf™ benchmark submissions: GitHub

Portable program pipeline (workflow)

Next we have implemented a CK module to provide a common API to compile, run, and validate programs while automatically adapting to any platform and environment:

A user describes dependencies on CK packages in the CK program meta as well as commands to build, pre-process, run, post-process, and validate a given program.

Examples:

ck pull repo:octoml@mlops

ck compile program:image-corner-detection --speed
ck run program:image-corner-detection --repeat=1 --env.OMP_NUM_THREADS=4

Reproducible experiments

We have developed an abstraction to record and reply experiments using the following CK module:

This module records all resolved dependencies, inputs and outputs when running above CK programs thus allowing to preserve experiments with all the provenance and replay them later on the same or different machine:

ck benchmark program:image-corner-detection --record --record_uoa=my_experiment

ck find experiment:my_experiment

ck replay experiment:my_experiment

ck zip experiment:my_experiment

Dashboards

Since we can record all experiments in a unified way, we can also visualize them in a unified way. That’s why we have developed a simple web server that can help to create customizable dashboards:

  • module:web [API]

See examples of such dashboards:

Interactive articles

One of our goals for CK was to automate the (re-)generation of reproducible articles. We have validated this possibility in this proof-of-concept project with the Raspberry Pi foundation.

We plan to develop a GUI to make the process of generating such papers more user friendly!

Jupyter notebooks

It is possible to use CK from Jupyter and Colab notebooks. We provided an abstraction to share Jupyter notebooks in CK repositories:

You can see an example of a Jupyter notebook with CK commands to process MLPerf™ benchmark results here.

Docker

We provided an abstraction to build, pull, and run Docker images:

You can see examples of Docker images with unified CK commands to automate the MLPerf™ benchmark here.

Further info

During the past few years we converted all the workflows and components from our past ML&systems R&D including the MILEPOST and cTuning.org project to the CK format.

There are now 150+ CK modules with actions automating and abstracting many tedious and repetitive tasks in ML&systems R&D including model training and prediction, universal autotuning, ML/SW/HW co-design, model testing and deployment, paper generation and so on:

Don’t hesitate to contact us if you have a feedback or want to know more about our plans!