Client#

class beaker.Beaker(config, check_for_upgrades=True, timeout=5.0, session=None, pool_maxsize=None, user_agent='beaker-py v1.26.6')[source]#

Bases: object

A client for interacting with Beaker.

Parameters:

  • config (Config) – The Beaker Config.

  • check_for_upgrades (bool, default: True) – Automatically check that beaker-py is up-to-date. You’ll see a warning if it isn’t.

  • timeout (Union[float, Tuple[float, float], None], default: 5.0) – How many seconds to wait for the Beaker server to send data before giving up, as a float, or a (connect timeout, read timeout) tuple.

  • session (Union[bool, Session, None], default: None) –

    Set to True or a requests.Session instance to force the Beaker client to use a single Session for all HTTP requests to the Beaker server for the life of the client.

    See also

    The session() context manager.

    Warning

    You should only set this argument for short-lived clients. If you’re initializing a Beaker client with this that’s supposed to stick around indefinitely, consider using the session() context manager intermittently instead.

  • pool_maxsize (Optional[int], default: None) – The maximum size of the connection pool to use. If not specified, a large default value will be used based on a multiple of the number of CPUs available.

  • user_agent (str, default: 'beaker-py v1.26.6') – Override the “User-Agent” header used in requests to the Beaker server.

The easiest way to initialize a Beaker client is with from_env():

>>> beaker = Beaker.from_env()

You can then interact with the various Beaker services through the corresponding property. For example, to manage workspaces, use Beaker.workspace:

>>> beaker.workspace.get(workspace_name).full_name
'ai2/beaker-py-testing'

Tip

Use the right side nav to browse through the API docs for all of the different services.

RECOVERABLE_SERVER_ERROR_CODES = (429, 500, 502, 503, 504)#
MAX_RETRIES = 5#
BACKOFF_FACTOR = 1#
BACKOFF_MAX = 120#
API_VERSION = 'v3'#
CLIENT_VERSION = '1.26.6'#
logger = <Logger beaker (WARNING)>#
classmethod from_env(check_for_upgrades=True, timeout=5.0, session=None, pool_maxsize=None, user_agent='beaker-py v1.26.6', **overrides)[source]#

Initialize client from a config file and/or environment variables.

Parameters:

  • check_for_upgrades (bool, default: True) – Automatically check that beaker-py is up-to-date. You’ll see a warning if it isn’t.

  • timeout (Union[float, Tuple[float, float], None], default: 5.0) – How many seconds to wait for the Beaker server to send data before giving up, as a float, or a (connect timeout, read timeout) tuple.

  • session (Union[bool, Session, None], default: None) –

    Set to True or a requests.Session instance to force the Beaker client to use a single Session for all HTTP requests to the Beaker server.

    See also

    The session() context manager.

    Warning

    You should only set this argument for short-lived clients. If you’re initializing a Beaker client with this that’s supposed to stick around indefinitely, consider using the session() context manager intermittently instead.

  • pool_maxsize (Optional[int], default: None) – The maximum size of the connection pool to use. If not specified, a large default value will be used based on a multiple of the number of CPUs available.

  • user_agent (str, default: 'beaker-py v1.26.6') – Override the “User-Agent” header used in requests to the Beaker server.

  • overrides – Fields in the Config to override.

Return type:

Beaker

Note

This will use the same config file that the Beaker command-line client creates and uses, which is usually located at $HOME/.beaker/config.yml.

If you haven’t configured the command-line client, then you can alternately just set the environment variable BEAKER_TOKEN to your Beaker user token.

session(session=None)[source]#

A context manager that forces the Beaker client to reuse a single requests.Session for all HTTP requests to the Beaker server.

This can improve performance when calling a series of a client methods in a row.

Examples:

>>> with beaker.session():
...     n_images = len(beaker.workspace.images())
...     n_datasets = len(beaker.workspace.datasets())
Parameters:

session (Optional[Session], default: None) –

The session to use. If not provided a default will be used.

Warning

Only set the session argument if you really know what you’re doing! Otherwise just leave this as None.

Return type:

Generator[None, None, None]

property config: Config#

The client’s Config.

property account: AccountClient#

Manage accounts.

Examples:

>>> beaker.account.name
'petew'

Tip

See the Accounts Overview for a walk-through of the main methods, or check out the Account API Docs to see all of the available methods.

property organization: OrganizationClient#

Manage organizations.

Examples:

>>> beaker.organization.get("ai2").display_name
'AI2'

Tip

See the Organizations Overview for a walk-through of the main methods, or check out the Organization API Docs to see all of the available methods.

property workspace: WorkspaceClient#

Manage workspaces.

Tip

See the Workspaces Overview for a walk-through of the main methods, or check out the Workspace API Docs to see all of the available methods.

property cluster: ClusterClient#

Manage clusters.

Examples:

>>> beaker.cluster.get(beaker_cluster_name).name
'ai2/canary'

Tip

See the Clusters Overview for a walk-through of the main methods, or check out the Cluster API Docs to see all of the available methods.

property node: NodeClient#

Manage nodes.

Examples:

>>> beaker.node.get(beaker_node_id).limits.gpu_count
8

Tip

See the Nodes Overview for a walk-through of the main methods, or check out the Node API Docs to see all of the available methods.

property dataset: DatasetClient#

Manage datasets.

Tip

See the Datasets Overview for a walk-through of the main methods, or check out the Dataset API Docs to see all of the available methods.

property image: ImageClient#

Manage images.

Examples:

>>> beaker.image.get("petew/hello-world").original_tag
'hello-world'

Tip

See the Images Overview for a walk-through of the main methods, or check out the Image API Docs to see all of the available methods.

property job: JobClient#

Manage jobs.

Examples:

>>> running_jobs = beaker.job.list(
...     beaker_on_prem_cluster_name,
...     finalized=False,
... )

Tip

See the Jobs Overview for a walk-through of the main methods, or check out the Job API Docs to see all of the available methods.

property experiment: ExperimentClient#

Manage experiments.

Examples:

>>> logs = "".join([
...     line.decode() for line in
...     beaker.experiment.logs("petew/hello-world", quiet=True)
... ])

Tip

See the Experiments Overview for a walk-through of the main methods, or check out the Experiment API Docs to see all of the available methods.

property secret: SecretClient#

Manage secrets.

Examples:

>>> secret = beaker.secret.write(secret_name, "foo")

Tip

See the Secrets Overview for a walk-through of the main methods, or check out the Secret API Docs to see all of the available methods.

property group: GroupClient#

Manage groups.

Examples:

>>> group = beaker.group.create(group_name)

Tip

See the Groups Overview for a walk-through of the main methods, or check out the Group API Docs to see all of the available methods.

property docker: DockerClient#

Account#

class beaker.services.AccountClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.account.

property name#
whoami()[source]#

Check who you are authenticated as.

Raises:
Return type:

Account

list_organizations()[source]#

List all organizations you are a member of.

Raises:
Return type:

List[Organization]

get(account)[source]#

Get information about an account.

Parameters:

account (str) – The account name or ID.

Raises:
Return type:

Account

Organization#

class beaker.services.OrganizationClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.organization.

get(org=None)[source]#

Get information about an organization.

Parameters:

org (Optional[str], default: None) – The organization name or ID. If not specified, Beaker.config.default_org is used.

Raises:
Return type:

Organization

add_member(account, org=None)[source]#

Add an account to an organization.

Parameters:
Raises:
Return type:

OrganizationMember

get_member(account, org=None)[source]#

Get information about an organization member.

Parameters:
Raises:
Return type:

OrganizationMember

list_members(org=None)[source]#

List members of an organization.

Parameters:

org (Union[str, Organization, None], default: None) – The organization name or object. If not specified, Beaker.config.default_org is used.

Raises:
Return type:

List[Account]

remove_member(account, org=None)[source]#

Remove a member from an organization.

Parameters:
Raises:

Workspace#

class beaker.services.WorkspaceClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.workspace.

get(workspace=None)[source]#

Get information about the workspace.

Parameters:

workspace (Optional[str], default: None) – The workspace name or ID. If not specified, Beaker.config.default_workspace is used.

Raises:
Return type:

Workspace

create(workspace, *, description=None, public=None)[source]#

Create a workspace.

Parameters:

  • workspace (str) – The workspace name.

  • description (Optional[str], default: None) – Text description for the workspace.

  • public (Optional[bool], default: None) – If the workspace should be public.

Raises:
Return type:

Workspace

ensure(workspace)[source]#

Ensure that the given workspace exists.

Parameters:

workspace (str) – The workspace name.

Raises:
Return type:

Workspace

archive(workspace)[source]#

Archive a workspace, making it read-only.

Parameters:

workspace (Union[str, Workspace]) – The workspace to archive.

Raises:
Return type:

Workspace

unarchive(workspace)[source]#

Unarchive a workspace.

Parameters:

workspace (Union[str, Workspace]) – The workspace to unarchive.

Raises:
Return type:

Workspace

rename(workspace, name)[source]#

Rename a workspace.

Parameters:

  • workspace (Union[str, Workspace]) – The workspace to rename.

  • name (str) – The new name to assign to the workspace. This should only not include the organization.

Raises:
Return type:

Workspace

move(*items, workspace=None)[source]#

Move items into a workspace.

Parameters:
Raises:
iter(org=None, *, author=None, match=None, archived=None, limit=None, sort_by=WorkspaceSort.created, descending=True, cursor=0)[source]#
Return type:

Generator[Workspace, None, None]

list(org=None, *, author=None, match=None, archived=None, limit=None, sort_by=WorkspaceSort.created, descending=True, cursor=0)[source]#

List workspaces belonging to an organization.

Parameters:

  • org (Union[str, Organization, None], default: None) – The organization name or object. If not specified, Beaker.config.default_org is used.

  • author (Union[str, Account, None], default: None) – Only list workspaces authored by this account.

  • match (Optional[str], default: None) – Only include workspaces matching the text.

  • archived (Optional[bool], default: None) – Only include/exclude archived workspaces.

  • limit (Optional[int], default: None) – Limit the number of workspaces returned.

  • sort_by (WorkspaceSort, default: <WorkspaceSort.created: 'created'>) – The field to sort the results by.

  • descending (bool, default: True) – Order the results in descending order according to the sort_by field.

  • cursor (int, default: 0) – Set the starting cursor for the query. You can use this to paginate the results.

Raises:
Return type:

List[Workspace]

iter_images(workspace=None, *, match=None, limit=None, sort_by=ImageSort.created, descending=True, cursor=0)[source]#

Iterate over the images in a workspace.

Parameters:

  • workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name or object. If not specified, Beaker.config.default_workspace is used.

  • match (Optional[str], default: None) – Only include images matching the text.

  • limit (Optional[int], default: None) – Limit the number of images returned.

  • sort_by (ImageSort, default: <ImageSort.created: 'created'>) – The field to sort the results by.

  • descending (bool, default: True) – Order the results in descending order according to the sort_by field.

  • cursor (int, default: 0) – Set the starting cursor for the query. You can use this to paginate the results.

Raises:
Return type:

Generator[Image, None, None]

images(workspace=None, *, match=None, limit=None, sort_by=ImageSort.created, descending=True, cursor=0)[source]#

List the images in a workspace.

Parameters:

  • workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name or object. If not specified, Beaker.config.default_workspace is used.

  • match (Optional[str], default: None) – Only include images matching the text.

  • limit (Optional[int], default: None) – Limit the number of images returned.

  • sort_by (ImageSort, default: <ImageSort.created: 'created'>) – The field to sort the results by.

  • descending (bool, default: True) – Order the results in descending order according to the sort_by field.

  • cursor (int, default: 0) – Set the starting cursor for the query. You can use this to paginate the results.

Raises:
Return type:

List[Image]

iter_experiments(workspace=None, *, match=None, limit=None, sort_by=ExperimentSort.created, descending=True, cursor=0)[source]#

Iterate over the experiments in a workspace.

Parameters:

  • workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name or object. If not specified, Beaker.config.default_workspace is used.

  • match (Optional[str], default: None) – Only include experiments matching the text.

  • limit (Optional[int], default: None) – Limit the number of experiments returned.

  • sort_by (ExperimentSort, default: <ExperimentSort.created: 'created'>) – The field to sort the results by.

  • descending (bool, default: True) – Order the results in descending order according to the sort_by field.

  • cursor (int, default: 0) – Set the starting cursor for the query. You can use this to paginate the results.

Raises:
Return type:

Generator[Experiment, None, None]

experiments(workspace=None, *, match=None, limit=None, sort_by=ExperimentSort.created, descending=True, cursor=0)[source]#

List the experiments in a workspace.

Parameters:

  • workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name or object. If not specified, Beaker.config.default_workspace is used.

  • match (Optional[str], default: None) – Only include experiments matching the text.

  • limit (Optional[int], default: None) – Limit the number of experiments returned.

  • sort_by (ExperimentSort, default: <ExperimentSort.created: 'created'>) – The field to sort the results by.

  • descending (bool, default: True) – Order the results in descending order according to the sort_by field.

  • cursor (int, default: 0) – Set the starting cursor for the query. You can use this to paginate the results.

Raises:
Return type:

List[Experiment]

iter_datasets(workspace=None, *, match=None, results=None, uncommitted=None, limit=None, sort_by=DatasetSort.created, descending=True, cursor=0)[source]#

Iterate over the datasets in a workspace.

Parameters:

  • workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name, or object. If not specified, Beaker.config.default_workspace is used.

  • match (Optional[str], default: None) – Only include datasets matching the text.

  • results (Optional[bool], default: None) – Only include/exclude experiment result datasets.

  • uncommitted (Optional[bool], default: None) – Only include/exclude uncommitted datasets.

  • limit (Optional[int], default: None) – Limit the number of datasets returned.

  • sort_by (DatasetSort, default: <DatasetSort.created: 'created'>) – The field to sort the results by.

  • descending (bool, default: True) – Order the results in descending order according to the sort_by field.

  • cursor (int, default: 0) – Set the starting cursor for the query. You can use this to paginate the results.

Raises:
Return type:

Generator[Dataset, None, None]

datasets(workspace=None, *, match=None, results=None, uncommitted=None, limit=None, sort_by=DatasetSort.created, descending=True, cursor=0)[source]#

List the datasets in a workspace.

Parameters:

  • workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name, or object. If not specified, Beaker.config.default_workspace is used.

  • match (Optional[str], default: None) – Only include datasets matching the text.

  • results (Optional[bool], default: None) – Only include/exclude experiment result datasets.

  • uncommitted (Optional[bool], default: None) – Only include/exclude uncommitted datasets.

  • limit (Optional[int], default: None) – Limit the number of datasets returned.

  • sort_by (DatasetSort, default: <DatasetSort.created: 'created'>) – The field to sort the results by.

  • descending (bool, default: True) – Order the results in descending order according to the sort_by field.

  • cursor (int, default: 0) – Set the starting cursor for the query. You can use this to paginate the results.

Raises:
Return type:

List[Dataset]

secrets(workspace=None)[source]#

List secrets in a workspace.

Parameters:

workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name, or object. If not specified, Beaker.config.default_workspace is used.

Raises:
Return type:

List[Secret]

iter_groups(workspace=None, *, match=None, limit=None, sort_by=GroupSort.created, descending=True, cursor=0)[source]#

Iterate over groups in a workspace.

Parameters:

  • workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name, or object. If not specified, Beaker.config.default_workspace is used.

  • match (Optional[str], default: None) – Only include groups matching the text.

  • limit (Optional[int], default: None) – Limit the number of groups returned.

  • sort_by (GroupSort, default: <GroupSort.created: 'created'>) – The field to sort the results by.

  • descending (bool, default: True) – Order the results in descending order according to the sort_by field.

  • cursor (int, default: 0) – Set the starting cursor for the query. You can use this to paginate the results.

Raises:
Return type:

Generator[Group, None, None]

groups(workspace=None, *, match=None, limit=None, sort_by=GroupSort.created, descending=True, cursor=0)[source]#

List groups in a workspace.

Parameters:

  • workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name, or object. If not specified, Beaker.config.default_workspace is used.

  • match (Optional[str], default: None) – Only include groups matching the text.

  • limit (Optional[int], default: None) – Limit the number of groups returned.

  • sort_by (GroupSort, default: <GroupSort.created: 'created'>) – The field to sort the results by.

  • descending (bool, default: True) – Order the results in descending order according to the sort_by field.

  • cursor (int, default: 0) – Set the starting cursor for the query. You can use this to paginate the results.

Raises:
Return type:

List[Group]

get_permissions(workspace=None)[source]#

Get workspace permissions.

Parameters:

workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name, or object. If not specified, Beaker.config.default_workspace is used.

Raises:
Return type:

WorkspacePermissions

grant_permissions(auth, *accounts, workspace=None)[source]#

Grant workspace permissions to accounts.

Parameters:
Raises:
Return type:

WorkspacePermissions

set_visibility(public=False, workspace=None)[source]#

Set workspace visibility to public or private.

Parameters:
Raises:
Return type:

WorkspacePermissions

revoke_permissions(*accounts, workspace=None)[source]#

Revoke workspace permissions to accounts.

Parameters:
Raises:
Return type:

WorkspacePermissions

url(workspace=None)[source]#

Get the URL for a workspace.

Parameters:

workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name, or object. If not specified, Beaker.config.default_workspace is used.

Raises:
Return type:

str

clear(workspace=None, *, groups=True, experiments=True, images=True, datasets=True, secrets=True, older_than=None)[source]#

Remove groups, experiments, images, datasets, and secrets from a workspace.

Parameters:

  • workspace (Union[str, Workspace, None], default: None) – The Beaker workspace name, or object. If not specified, Beaker.config.default_workspace is used.

  • groups (bool, default: True) – Whether to delete groups.

  • experiments (bool, default: True) – Whether to delete experiments.

  • images (bool, default: True) – Whether to delete images.

  • datasets (bool, default: True) – Whether to delete datasets.

  • secrets (bool, default: True) – Whether to delete secrets.

  • older_than (Optional[datetime], default: None) – Only delete objects created before this date.

Raises:
Return type:

WorkspaceClearResult

Cluster#

class beaker.services.ClusterClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.cluster.

get(cluster)[source]#

Get information about the cluster.

Parameters:

cluster (str) – The cluster name or ID.

Raises:
Return type:

Cluster

create(name, max_size=1, preemptible=False, cpus=None, gpus=0, gpu_type=None, memory=None)[source]#

Create a new Beaker cloud cluster.

Note

For creating on-premise clusters you should still use the Beaker CLI.

Parameters:

  • name (str) – The name to assign to the new cluster. If Config.default_org is not set, the name should start with the name of an organization: “{organization}/{cluster_name}”, e.g. “ai2/my-new-cluster”.

  • max_size (int, default: 1) – The maximum number of nodes the cluster can scale up to.

  • preemptible (bool, default: False) – Use preemptible cloud machines for the nodes.

  • cpus (Optional[float], default: None) – The number of virtual CPU available to each node.

  • gpus (int, default: 0) – The number of GPUs available to each node.

  • gpu_type (Optional[str], default: None) – The type of GPU available to each node.

  • memory (Optional[str], default: None) – The amount of memory available to each node, specified as a number with a unit suffix. E.g. “2.5GiB”.

Raises:
Return type:

Cluster

update(cluster, max_size=None, allow_preemptible=None)[source]#

Modify a cluster.

Parameters:

  • cluster (Union[str, Cluster]) – The cluster ID, full name, or object.

  • max_size (Optional[int], default: None) – The maximum number of nodes.

  • allow_preemptible (Optional[bool], default: None) – Allow or disallow preemptible jobs.

Raises:
Return type:

Cluster

delete(cluster)[source]#

Delete a cluster.

Parameters:

cluster (Union[str, Cluster]) – The cluster ID, full name, or object.

Raises:
list(org=None)[source]#

List clusters under an organization.

Parameters:

org (Union[str, Organization, None], default: None) – The organization name or object. If not specified, Beaker.config.default_org is used.

Raises:
Return type:

List[Cluster]

nodes(cluster)[source]#

List the nodes in a cluster.

Parameters:

cluster (Union[str, Cluster]) – The cluster ID, full name, or object.

Raises:
Return type:

List[Node]

utilization(cluster)[source]#

Get current utilization stats for each node in a cluster.

Parameters:

cluster (Union[str, Cluster]) – The cluster ID, full name, or object.

Raises:
Return type:

ClusterUtilization

filter_available(resources, *clusters)[source]#

Filter out clusters that don’t have enough available resources, returning a list of ClusterUtilization for each cluster that has sufficient resources.

This can be used, for example, to automatically find an on-premise cluster with enough free resources to run a particular task.

Caution

This method is experimental and may change or be removed in future releases.

Parameters:
Raises:
Return type:

List[ClusterUtilization]

url(cluster)[source]#

Get the URL for a cluster.

Parameters:

cluster (Union[str, Cluster]) – The cluster ID, full name, or object.

Raises:

ClusterNotFound – If the cluster doesn’t exist.

Return type:

str

preempt_jobs(cluster, ignore_failures=False)[source]#

Preempt all preemptible jobs on the cluster.

Parameters:

  • cluster (Union[str, Cluster]) – The cluster ID, full name, or object.

  • ignore_failures (bool, default: False) – If True, any jobs that fail to preempt will be ignored.

Raises:
Return type:

List[Job]

Node#

class beaker.services.NodeClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.node.

get(node_id)[source]#

Get information about a node.

Parameters:

node_id (str) – The ID of the node.

Raises:
Return type:

Node

Dataset#

class beaker.services.DatasetClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.dataset.

HEADER_UPLOAD_ID = 'Upload-ID'#
HEADER_UPLOAD_LENGTH = 'Upload-Length'#
HEADER_UPLOAD_OFFSET = 'Upload-Offset'#
HEADER_DIGEST = 'Digest'#
HEADER_LAST_MODIFIED = 'Last-Modified'#
HEADER_CONTENT_LENGTH = 'Content-Length'#
REQUEST_SIZE_LIMIT: ClassVar[int] = 33554432#
DOWNLOAD_CHUNK_SIZE: ClassVar[int] = 10240#

The default buffer size for downloads.

get(dataset)[source]#

Get info about a dataset.

Parameters:

dataset (str) – The dataset ID or name.

Raises:
Return type:

Dataset

create(name, *sources, target=None, workspace=None, description=None, force=False, max_workers=None, quiet=False, commit=True, strip_paths=False)[source]#

Create a dataset with the source file(s).

Parameters:

  • name (str) – The name to assign to the new dataset.

  • sources (Union[PathLike, str]) – Local source files or directories to upload to the dataset.

  • target (Union[PathLike, str, None], default: None) – If specified, all source files/directories will be uploaded under a directory of this name.

  • workspace (Optional[str], default: None) – The workspace to upload the dataset to. If not specified, Beaker.config.default_workspace is used.

  • description (Optional[str], default: None) – Text description for the dataset.

  • force (bool, default: False) – If True and a dataset by the given name already exists, it will be overwritten.

  • max_workers (Optional[int], default: None) – The maximum number of thread pool workers to use to upload files concurrently.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • commit (bool, default: True) – Whether to commit the dataset after successful upload.

  • strip_paths (bool, default: False) –

    If True, all source files and directories will be uploaded under their name, not their path. E.g. the file “docs/source/index.rst” would be uploaded as just “index.rst”, instead of “docs/source/index.rst”.

    Note

    This only applies to source paths that are children of the current working directory. If a source path is outside of the current working directory, it will always be uploaded under its name only.

Raises:
Return type:

Dataset

commit(dataset)[source]#

Commit the dataset.

Parameters:

dataset (Union[str, Dataset]) – The dataset ID, name, or object.

Raises:
Return type:

Dataset

fetch(dataset, target=None, prefix=None, force=False, max_workers=None, quiet=False, validate_checksum=True, chunk_size=None)[source]#

Download a dataset.

Parameters:

  • dataset (Union[str, Dataset]) – The dataset ID, name, or object.

  • target (Union[PathLike, str, None], default: None) – The target path to download fetched data to. Defaults to Path(.).

  • prefix (Optional[str], default: None) – Only download files that start with the given prefix.

  • max_workers (Optional[int], default: None) – The maximum number of thread pool workers to use to download files concurrently.

  • force (bool, default: False) – If True, existing local files will be overwritten.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • validate_checksum (bool, default: True) – If True, the checksum of every file downloaded will be verified.

  • chunk_size (Optional[int], default: None) – The size of the buffer (in bytes) to use while downloading each file. Defaults to DOWNLOAD_CHUNK_SIZE.

Raises:
stream_file(dataset, file, offset=0, length=-1, quiet=False, validate_checksum=True, chunk_size=None)[source]#

Stream download the contents of a single file from a dataset.

See also

get_file() is similar but returns the entire contents at once instead of a generator over the contents.

Parameters:

  • dataset (Union[str, Dataset]) – The dataset ID, name, or object.

  • file (Union[str, FileInfo]) – The path of the file within the dataset or the corresponding FileInfo object.

  • offset (int, default: 0) – Offset to start from, in bytes.

  • length (int, default: -1) – Number of bytes to read.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • validate_checksum (bool, default: True) – If True, the checksum of the downloaded bytes will be verified.

  • chunk_size (Optional[int], default: None) – The size of the buffer (in bytes) to use while downloading each file. Defaults to DOWNLOAD_CHUNK_SIZE.

Raises:
Examples:

>>> total_bytes = 0
:rtype: :py:class:`~typing.Generator`\[:py:class:`bytes`, :py:obj:`None`, :py:obj:`None`]
>>> with open(tmp_path / squad_dataset_file_name, "wb") as f:
...     for chunk in beaker.dataset.stream_file(squad_dataset_name, squad_dataset_file_name, quiet=True):
...         total_bytes += f.write(chunk)
get_file(dataset, file, offset=0, length=-1, quiet=False, validate_checksum=True, chunk_size=None)[source]#

Download the contents of a single file from a dataset.

See also

stream_file() is similar but returns a generator over the contents.

Parameters:

  • dataset (Union[str, Dataset]) – The dataset ID, name, or object.

  • file (Union[str, FileInfo]) – The path of the file within the dataset or the corresponding FileInfo object.

  • offset (int, default: 0) – Offset to start from, in bytes.

  • length (int, default: -1) – Number of bytes to read.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • validate_checksum (bool, default: True) – If True, the checksum of the downloaded bytes will be verified.

  • chunk_size (Optional[int], default: None) – The size of the buffer (in bytes) to use while downloading each file. Defaults to DOWNLOAD_CHUNK_SIZE.

Raises:
Examples:

Return type:

bytes

>>> contents = beaker.dataset.get_file(squad_dataset_name, squad_dataset_file_name, quiet=True)
file_info(dataset, file_name)[source]#

Get the FileInfo for a file in a dataset.

Parameters:

  • dataset (Union[str, Dataset]) – The dataset ID, name, or object.

  • file_name (str) – The path of the file within the dataset.

Raises:
Return type:

FileInfo

delete(dataset)[source]#

Delete a dataset.

Parameters:

dataset (Union[str, Dataset]) – The dataset ID, name, or object.

Raises:
sync(dataset, *sources, target=None, quiet=False, max_workers=None, strip_paths=False)[source]#

Sync local files or directories to an uncommitted dataset.

Parameters:

  • dataset (Union[str, Dataset]) – The dataset ID, name, or object.

  • sources (Union[PathLike, str]) – Local source files or directories to upload to the dataset.

  • target (Union[PathLike, str, None], default: None) – If specified, all source files/directories will be uploaded under a directory of this name.

  • max_workers (Optional[int], default: None) – The maximum number of thread pool workers to use to upload files concurrently.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • strip_paths (bool, default: False) –

    If True, all source files and directories will be uploaded under their name, not their path. E.g. the file “docs/source/index.rst” would be uploaded as just “index.rst”, instead of “docs/source/index.rst”.

    Note

    This only applies to source paths that are children of the current working directory. If a source path is outside of the current working directory, it will always be uploaded under its name only.

Raises:
Return type:

None

upload(dataset, source, target, quiet=False)[source]#

Upload raw bytes to an uncommitted dataset.

Parameters:

  • dataset (Union[str, Dataset]) – The dataset ID, name, or object.

  • source (bytes) – The raw bytes to upload to the dataset.

  • target (Union[PathLike, str]) – The name to assign to the file for the bytes in the dataset.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

Raises:
Return type:

None

ls(dataset, prefix=None)[source]#

List files in a dataset.

Parameters:

  • dataset (Union[str, Dataset]) – The dataset ID, name, or object.

  • prefix (Optional[str], default: None) – An optional path prefix to filter by.

Raises:
Return type:

List[FileInfo]

size(dataset)[source]#

Calculate the size of a dataset, in bytes.

Parameters:

dataset (Union[str, Dataset]) – The dataset ID, name, or object.

Raises:
Return type:

int

rename(dataset, name)[source]#

Rename a dataset.

Parameters:

  • dataset (Union[str, Dataset]) – The dataset ID, name, or object.

  • name (str) – The new name of the dataset.

Raises:
Return type:

Dataset

url(dataset)[source]#

Get the URL for a dataset.

Parameters:

dataset (Union[str, Dataset]) – The dataset ID, name, or object.

Raises:

DatasetNotFound – If the dataset can’t be found.

Return type:

str

Image#

class beaker.services.ImageClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.image.

get(image)[source]#

Get info about an image on Beaker.

Parameters:

image (str) – The Beaker image ID or name.

Raises:

  • ImageNotFound – If the image can’t be found on Beaker.

  • RequestException – Any other exception that can occur when contacting the Beaker server.

Return type:

Image

create(name, image_tag, workspace=None, description=None, quiet=False, commit=True)[source]#

Upload a Docker image to Beaker.

Parameters:

  • name (str) – The name to assign to the image on Beaker.

  • image_tag (str) – The tag of the local image you’re uploading.

  • workspace (Union[str, Workspace, None], default: None) – The workspace to upload the image to. If not specified, Beaker.config.default_workspace is used.

  • description (Optional[str], default: None) – Text description of the image.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • commit (bool, default: True) – Whether to commit the image after successful upload.

Raises:
Return type:

Image

commit(image)[source]#

Commit an image.

Parameters:

image (Union[str, Image]) – The Beaker image ID, name, or object.

Raises:
Return type:

Image

delete(image)[source]#

Delete an image on Beaker.

Parameters:

image (Union[str, Image]) – The Beaker image ID, name, or object.

Raises:
rename(image, name)[source]#

Rename an image on Beaker.

Parameters:

  • image (Union[str, Image]) – The Beaker image ID, name, or object.

  • name (str) – The new name for the image.

Raises:
Return type:

Image

pull(image, quiet=False)[source]#

Pull an image from Beaker.

Important

This method returns a Docker Image, not a Beaker Image.

Parameters:

  • image (Union[str, Image]) – The Beaker image ID, name, or object.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

Raises:
Return type:

Image

url(image)[source]#

Get the URL for an image.

Parameters:

image (Union[str, Image]) – The Beaker image ID, name, or object.

Raises:

ImageNotFound – If the image can’t be found on Beaker.

Return type:

str

Job#

class beaker.services.JobClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.job.

get(job_id)[source]#

Get information about a job.

Parameters:

job_id (str) – The ID of the Beaker job.

Raises:
Return type:

Job

list(cluster=None, experiment=None, finalized=False, kind=None, node=None)[source]#

List jobs.

Parameters:

  • cluster (Union[str, Cluster, None], default: None) – List jobs on a cluster.

  • experiment (Union[str, Experiment, None], default: None) – List jobs in an experiment.

  • finalized (bool, default: False) – List only finalized or non-finalized jobs.

  • kind (Optional[JobKind], default: None) – List jobs of a certain kind.

  • node (Union[str, Node, None], default: None) – List jobs on a node.

Return type:

List[Job]

Important

Either cluster, experiment, or node must be specified. If node is specified, neither cluster nor experiment can be specified.

Raises:
logs(job, quiet=False, since=None)[source]#

Download the logs for a job.

Returns a generator with the streaming bytes from the download. The generator should be exhausted, otherwise the logs downloaded will be incomplete.

See also

Beaker.experiment.logs()

See also

follow()

Parameters:

  • job (Union[str, Job]) – The Beaker job ID or object.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • since (Union[str, datetime, timedelta, None], default: None) – Only show logs since a particular time. Could be a datetime object (naive datetimes will be treated as UTC), a timestamp string in the form of RFC 3339 (e.g. “2013-01-02T13:23:37Z”), or a relative time (e.g. a timedelta or a string like “42m”).

Raises:
Return type:

Generator[bytes, None, None]

metrics(job)[source]#

Get the metrics from a job.

See also

Beaker.experiment.metrics()

Parameters:

job (Union[str, Job]) – The Beaker job ID or object.

Raises:
Return type:

Optional[Dict[str, Any]]

results(job)[source]#

Get the results from a job.

See also

Beaker.experiment.results()

Parameters:

job (Union[str, Job]) – The Beaker job ID or object.

Raises:
Return type:

Optional[Dataset]

finalize(job)[source]#

Finalize a job.

Parameters:

job (Union[str, Job]) – The Beaker job ID or object.

Raises:
Return type:

Job

preempt(job)[source]#

Preempt a job.

Parameters:

job (Union[str, Job]) – The Beaker job ID or object.

Raises:
Return type:

Job

stop(job)[source]#

Stop a job.

Parameters:

job (Union[str, Job]) – The Beaker job ID or object.

Raises:
Return type:

Job

wait_for(*jobs, timeout=None, poll_interval=1.0, quiet=False, strict=False)[source]#

Wait for jobs to finalize, returning the completed jobs as a list in the same order they were given as input.

Caution

This method is experimental and may change or be removed in future releases.

See also

as_completed()

See also

follow()

See also

Beaker.experiment.wait_for()

Parameters:

  • jobs (Union[str, Job]) – Job ID, name, or object.

  • timeout (Optional[float], default: None) – Maximum amount of time to wait for (in seconds).

  • poll_interval (float, default: 1.0) – Time to wait between polling each job’s status (in seconds).

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • strict (bool, default: False) – If True, the exit code of each job will be checked, and a JobFailedError will be raised for non-zero exit codes.

Raises:
Return type:

List[Job]

as_completed(*jobs, timeout=None, poll_interval=1.0, quiet=False, strict=False)[source]#

Wait for jobs to finalize, returning an iterator that yields jobs as they complete.

Caution

This method is experimental and may change or be removed in future releases.

See also

wait_for()

See also

follow()

See also

Beaker.experiment.as_completed()

Parameters:

  • jobs (Union[str, Job]) – Job ID, name, or object.

  • timeout (Optional[float], default: None) – Maximum amount of time to wait for (in seconds).

  • poll_interval (float, default: 1.0) – Time to wait between polling each job’s status (in seconds).

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • strict (bool, default: False) – If True, the exit code of each job will be checked, and a JobFailedError will be raised for non-zero exit codes.

Raises:
Return type:

Generator[Job, None, None]

follow(job, timeout=None, strict=False, include_timestamps=True)[source]#

Follow a job live, creating a generator that produces log lines (as bytes) from the job as they become available. The return value of the generator is the finalized Job object.

See also

logs()

See also

wait_for()

See also

as_completed()

See also

Beaker.experiment.follow()

Parameters:

  • job (Union[str, Job]) – Job ID, name, or object.

  • timeout (Optional[float], default: None) – Maximum amount of time to follow job for (in seconds).

  • strict (bool, default: False) – If True, the exit code of each job will be checked, and a JobFailedError will be raised for non-zero exit codes.

  • include_timestamps (bool, default: True) – If True (the default) timestamps from the Beaker logs will be included in the output.

Raises:
Examples:

>>> job = beaker.experiment.latest_job(hello_world_experiment_name)
>>> for line in beaker.job.follow(job):
...     # Every log line from Beaker starts with an RFC 3339 UTC timestamp
...     # (e.g. '2021-12-07T19:30:24.637600011Z'). If we don't want to print
...     # the timestamps we can split them off like this:
...     line = line[line.find(b"Z ")+2:]
...     print(line.decode(errors="ignore"), end="")

Hello from Docker!
:rtype: :py:class:`~typing.Generator`\[:py:class:`bytes`, :py:obj:`None`, :py:class:`~beaker.data_model.job.Job`]
This message shows that your installation appears to be working correctly.

...
url(job)[source]#
Return type:

str

Experiment#

class beaker.services.ExperimentClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.experiment.

get(experiment)[source]#

Get info about an experiment.

Parameters:

experiment (str) – The experiment ID or name.

Raises:
Return type:

Experiment

create(*args, **kwargs)[source]#

Create a new Beaker experiment with the given spec.

Parameters:
Raises:
Return type:

Experiment

spec(experiment)[source]#

Get the spec of an experiment.

Parameters:

experiment (Union[str, Experiment]) – The experiment ID, name, or object.

Raises:
Return type:

ExperimentSpec

stop(experiment)[source]#

Stop an experiment.

Parameters:

experiment (Union[str, Experiment]) – The experiment ID, name, or object.

Raises:
resume(experiment)[source]#

Resume a preempted experiment.

Parameters:

experiment (Union[str, Experiment]) – The experiment ID, name, or object.

Raises:
delete(experiment, delete_results_datasets=True)[source]#

Delete an experiment.

Parameters:

  • experiment (Union[str, Experiment]) – The experiment ID, name, or object.

  • delete_results_datasets (bool, default: True) – Also delete the experiment’s results datasets.

Raises:
rename(experiment, name)[source]#

Rename an experiment.

Parameters:

  • experiment (Union[str, Experiment]) – The experiment ID, name, or object.

  • name (str) – The new name for the experiment.

Raises:
Return type:

Experiment

tasks(experiment)[source]#

List the tasks in an experiment.

Parameters:

experiment (Union[str, Experiment]) – The experiment ID, name, or object.

Raises:
Examples:

Return type:

Tasks

>>> task = beaker.experiment.tasks(hello_world_experiment_name)["main"]
logs(experiment, task=None, quiet=False, since=None)[source]#

Download the logs for an experiment.

Returns a generator with the streaming bytes from the download. The generator should be exhausted, otherwise the logs downloaded will be incomplete.

Important

When there are multiple jobs for the given experiment / task, the logs for the latest job will be returned.

See also

Beaker.job.logs()

Parameters:

  • experiment (Union[str, Experiment]) – The experiment ID, name, or object.

  • task (Union[str, Task, None], default: None) – The task ID, name, or object of a specific task from the Beaker experiment to fetch logs for. Required if there are multiple tasks in the experiment.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • since (Union[str, datetime, timedelta, None], default: None) – Only show logs since a particular time. Could be a datetime object (naive datetimes will be treated as UTC), a timestamp string in the form of RFC 3339 (e.g. “2013-01-02T13:23:37Z”), or a relative time (e.g. a timedelta or a string like “42m”).

Raises:
Return type:

Generator[bytes, None, None]

metrics(experiment, task=None)[source]#

Get the metrics from a task in an experiment.

Important

When there are multiple jobs for the given experiment / task, the metrics for the latest finalized job will be returned.

See also

Beaker.job.metrics()

Parameters:

  • experiment (Union[str, Experiment]) – The experiment ID, name, or object.

  • task (Union[str, Task, None], default: None) – The task ID, name, or object of a specific task from the Beaker experiment to fetch metrics for. Required if there are multiple tasks in the experiment.

Raises:
Return type:

Optional[Dict[str, Any]]

results(experiment, task=None)[source]#

Get the result dataset from a task in an experiment.

Important

When there are multiple jobs for the given experiment / task, the results for dataset the latest finalized job will be returned.

See also

Beaker.job.results()

Parameters:

  • experiment (Union[str, Experiment]) – The experiment ID, name, or object.

  • task (Union[str, Task, None], default: None) – The task ID, name, or object of a specific task from the Beaker experiment to fetch results for. Required if there are multiple tasks in the experiment.

Raises:
Return type:

Optional[Dataset]

wait_for(*experiments, timeout=None, poll_interval=1.0, quiet=False, strict=False)[source]#

Wait for experiments to finalize, returning the completed experiments as a list in the same order they were given as input.

Caution

This method is experimental and may change or be removed in future releases.

See also

as_completed()

See also

follow()

See also

Beaker.job.wait_for()

Parameters:

  • experiments (Union[str, Experiment]) – Experiment ID, name, or object.

  • timeout (Optional[float], default: None) – Maximum amount of time to wait for (in seconds).

  • poll_interval (float, default: 1.0) – Time to wait between polling the experiment (in seconds).

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • strict (bool, default: False) – If True, the exit code of each job will be checked, and a JobFailedError will be raised for non-zero exit codes.

Raises:
Return type:

List[Experiment]

as_completed(*experiments, timeout=None, poll_interval=1.0, quiet=False, strict=False)[source]#

Wait for experiments to finalize, returning an iterator that yields experiments as they complete.

Caution

This method is experimental and may change or be removed in future releases.

See also

wait_for()

See also

follow()

See also

Beaker.job.as_completed()

Parameters:

  • experiments (Union[str, Experiment]) – Experiment ID, name, or object.

  • timeout (Optional[float], default: None) – Maximum amount of time to wait for (in seconds).

  • poll_interval (float, default: 1.0) – Time to wait between polling the experiment (in seconds).

  • quiet (bool, default: False) – If True, progress won’t be displayed.

  • strict (bool, default: False) – If True, the exit code of each job will be checked, and a JobFailedError will be raised for non-zero exit codes.

Raises:
Return type:

Generator[Experiment, None, None]

follow(experiment, task=None, timeout=None, strict=False, include_timestamps=True)[source]#

Follow an experiment live, creating a generator that produces log lines (as bytes) from the task’s job as they become available. The return value of the generator is the final Experiment object.

See also

logs()

See also

wait_for()

See also

as_completed()

See also

Beaker.job.follow()

Parameters:

  • experiment (Union[str, Experiment]) – Experiment ID, name, or object.

  • task (Union[str, Task, None], default: None) – The task ID, name, or object of a specific task from the Beaker experiment to follow. Required if there are multiple tasks in the experiment.

  • timeout (Optional[float], default: None) – Maximum amount of time to wait for (in seconds).

  • strict (bool, default: False) – If True, the exit code of the job will be checked, and a JobFailedError will be raised for non-zero exit codes.

  • include_timestamps (bool, default: True) – If True (the default) timestamps from the Beaker logs will be included in the output.

Raises:
Examples:

>>> for line in beaker.experiment.follow(hello_world_experiment_name):
...     # Every log line from Beaker starts with an RFC 3339 UTC timestamp
...     # (e.g. '2021-12-07T19:30:24.637600011Z'). If we don't want to print
...     # the timestamps we can split them off like this:
...     line = line[line.find(b"Z ")+2:]
...     print(line.decode(errors="ignore"), end="")

Hello from Docker!
:rtype: :py:class:`~typing.Generator`\[:py:class:`bytes`, :py:obj:`None`, :py:class:`~beaker.data_model.experiment.Experiment`]
This message shows that your installation appears to be working correctly.

...
url(experiment, task=None)[source]#

Get the URL for an experiment.

Parameters:

  • experiment (Union[str, Experiment]) – The experiment ID, name, or object.

  • task (Union[str, Task, None], default: None) – The task ID, name, or object of a specific task from the Beaker experiment to get the url for.

Raises:
Return type:

str

latest_job(experiment, task=None, ensure_finalized=False)[source]#

Get the latest job that ran for a task in an experiment.

Parameters:

  • experiment (Union[str, Experiment]) – The experiment ID, name, or object.

  • task (Union[str, Task, None], default: None) – The take ID, name, or object.

  • ensure_finalized (bool, default: False) – Consider only finalized jobs.

Raises:
Return type:

Optional[Job]

Secret#

class beaker.services.SecretClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.secret.

get(secret, workspace=None)[source]#

Get metadata about a secret.

Parameters:
Raises:
Return type:

Secret

read(secret, workspace=None)[source]#

Read the value of a secret.

Parameters:
Raises:
Return type:

str

write(name, value, workspace=None)[source]#

Write a new secret or update an existing one.

Parameters:
Raises:
Return type:

Secret

delete(secret, workspace=None)[source]#

Permanently delete a secret.

Parameters:
Raises:

Group#

class beaker.services.GroupClient(beaker)[source]#

Bases: ServiceClient

Accessed via Beaker.group.

get(group)[source]#

Get info about a group.

Parameters:

group (str) – The group ID or name.

Raises:
Return type:

Group

create(name, *experiments, description=None, workspace=None)[source]#
Parameters:
Raises:
Return type:

Group

delete(group)[source]#

Delete a group.

Parameters:

group (Union[str, Group]) – The group ID, name, or object.

Raises:
rename(group, name)[source]#

Rename a group.

Parameters:

  • group (Union[str, Group]) – The group ID, name, or object.

  • name (str) – The new name for the group.

Raises:
Return type:

Group

add_experiments(group, *experiments)[source]#

Add experiments to a group.

Parameters:
Raises:
remove_experiments(group, *experiments)[source]#

Remove experiments from a group.

Parameters:
Raises:
list_experiments(group)[source]#

List experiments in a group.

Parameters:

group (Union[str, Group]) – The group ID, name, or object.

Raises:
Return type:

List[Experiment]

export_experiments(group, quiet=False)[source]#

Export all experiments and metrics in a group as a CSV.

Returns a generator that should be exhausted to get the complete file.

Parameters:

  • group (Union[str, Group]) – The group ID, name, or object.

  • quiet (bool, default: False) – If True, progress won’t be displayed.

Raises:
Return type:

Generator[bytes, None, None]

url(group)[source]#

Get the URL for a group.

Parameters:

group (Union[str, Group]) – The group ID, name, or object.

Raises:

GroupNotFound – If the group can’t be found.

Return type:

str