ahriman.core.upload package

Submodules

ahriman.core.upload.github module

class GitHub(repository_id: RepositoryId, configuration: Configuration, section: str)

Bases: Upload, HttpUpload

upload files to GitHub releases

github_owner

GitHub repository owner

Type:

str

github_release_tag

GitHub release tag

Type:

str

github_release_tag_name

GitHub release tag name

Type:

str

github_repository

GitHub repository name

Type:

str

default constructor

Parameters:

  • repository_id (RepositoryId) – repository unique identifier

  • configuration (Configuration) – configuration instance

  • section (str) – settings section name

asset_remove(release: dict[str, Any], name: str) → None

remove asset from the release by name

Parameters:

  • release (dict[str, Any]) – release object

  • name (str) – asset name

asset_upload(release: dict[str, Any], path: Path) → None

upload asset to the release

Parameters:

  • release (dict[str, Any]) – release object

  • path (Path) – path to local file

files_remove(release: dict[str, Any], local_files: dict[Path, str], remote_files: dict[str, str]) → None

remove files from GitHub

Parameters:

  • release (dict[str, Any]) – release object

  • local_files (dict[Path, str]) – map of local file paths to its checksum

  • remote_files (dict[str, str]) – map of the remote files and its checksum

files_upload(release: dict[str, Any], local_files: dict[Path, str], remote_files: dict[str, str]) → None

upload files to GitHub

Parameters:

  • release (dict[str, Any]) – release object

  • local_files (dict[Path, str]) – map of local file paths to its checksum

  • remote_files (dict[str, str]) – map of the remote files and its checksum

get_local_files(path: Path) → dict[Path, str]

get all local files and their calculated checksums

Parameters:

path (Path) – local path to sync

Returns:

map of path objects to its checksum

Return type:

dict[Path, str]

release_create() → dict[str, Any]

create empty release

Returns:

GitHub API release object for the new release

Return type:

dict[str, Any]

release_get() → dict[str, Any] | None

get release object if any

Returns:

GitHub API release object if release found and None otherwise

Return type:

dict[str, Any] | None

release_update(release: dict[str, Any], body: str) → None

update release

Parameters:

  • release (dict[str, Any]) – release object

  • body (str) – new release body

sync(path: Path, built_packages: list[Package]) → None

sync data to remote server

Parameters:

  • path (Path) – local path to sync

  • built_packages (list[Package]) – list of packages which has just been built

ahriman.core.upload.http_upload module

class HttpUpload(configuration: Configuration | None = None, section: str | None = None, *, suppress_errors: bool = False)

Bases: SyncHttpClient

helper for the http based uploads

default constructor

Parameters:

  • configuration (Configuration | None, optional) – configuration instance (Default value = None)

  • section (str | None, optional) – settings section name (Default value = None)

  • suppress_errors (bool, optional) – suppress logging of request errors (Default value = False)

static calculate_hash(path: Path) → str

calculate file checksum

Parameters:

path (Path) – path to local file

Returns:

calculated checksum of the file

Return type:

str

static get_body(local_files: dict[Path, str]) → str

generate release body from the checksums as returned from HttpUpload.get_hashes method

Parameters:

local_files (dict[Path, str]) – map of the paths to its checksum

Returns:

body to be inserted into release

Return type:

str

static get_hashes(body: str) → dict[str, str]

get checksums of the content from the repository

Parameters:

body (str) – release string body object

Returns:

map of the filename to its checksum as it is written in body

Return type:

dict[str, str]

ahriman.core.upload.remote_service module

class RemoteService(repository_id: RepositoryId, configuration: Configuration, section: str)

Bases: Upload, HttpUpload

upload files to another server instance

client

web client instance

Type:

WebClient

default constructor

Parameters:

  • repository_id (RepositoryId) – repository unique identifier

  • configuration (Configuration) – configuration instance

  • section (str) – settings section name

package_upload(path: Path, package: Package) → None

upload single package to remote

Parameters:

  • path (Path) – local path to sync

  • package (Package) – package to upload

sync(path: Path, built_packages: list[Package]) → None

sync data to remote server

Parameters:

  • path (Path) – local path to sync

  • built_packages (list[Package]) – list of packages which has just been built

property session: Session

get or create session

Returns:

created session object

Return type:

request.Session

ahriman.core.upload.rsync module

class Rsync(repository_id: RepositoryId, configuration: Configuration, section: str)

Bases: Upload

rsync wrapper

command

command arguments for sync

Type:

list[str]

remote

remote address to sync

Type:

str

default constructor

Parameters:

  • repository_id (RepositoryId) – repository unique identifier

  • configuration (Configuration) – configuration instance

  • section (str) – settings section name

sync(path: Path, built_packages: list[Package]) → None

sync data to remote server

Parameters:

  • path (Path) – local path to sync

  • built_packages (list[Package]) – list of packages which has just been built

ahriman.core.upload.s3 module

class S3(repository_id: RepositoryId, configuration: Configuration, section: str)

Bases: Upload

boto3 wrapper

Attributes

bucket(Any): boto3 S3 bucket object chunk_size(int): chunk size for calculating checksums object_path(Path): relative path to which packages will be uploaded

default constructor

Parameters:

  • repository_id (RepositoryId) – repository unique identifier

  • configuration (Configuration) – configuration instance

  • section (str) – settings section name

static calculate_etag(path: Path, chunk_size: int) → str

calculate amazon s3 etag credits to https://teppen.io/2018/10/23/aws_s3_verify_etags/ For this method we have to define nosec because it is out of any security context and provided by AWS

Parameters:

  • path (Path) – path to local file

  • chunk_size (int) – read chunk size, which depends on client settings

Returns:

calculated entity tag for local file

Return type:

str

static files_remove(local_files: dict[Path, str], remote_objects: dict[Path, Any]) → None

remove files which have been removed locally

Parameters:

  • local_files (dict[Path, str]) – map of local path object to its checksum

  • remote_objects (dict[Path, Any]) – map of remote path object to the remote s3 object

files_upload(path: Path, local_files: dict[Path, str], remote_objects: dict[Path, Any]) → None

upload changed files to s3

Parameters:

  • path (Path) – local path to sync

  • local_files (dict[Path, str]) – map of local path object to its checksum

  • remote_objects (dict[Path, Any]) – map of remote path object to the remote s3 object

static get_bucket(configuration: Configuration, section: str) → Any

create resource client from configuration

Parameters:

  • configuration (Configuration) – configuration instance

  • section (str) – settings section name

Returns:

amazon client

Return type:

Any

get_local_files(path: Path) → dict[Path, str]

get all local files and their calculated checksums

Parameters:

path (Path) – local path to sync

Returns:

map of path object to its checksum

Return type:

dict[Path, str]

get_remote_objects() → dict[Path, Any]

get all remote objects and their checksums

Returns:

map of path object to the remote s3 object

Return type:

dict[Path, Any]

sync(path: Path, built_packages: list[Package]) → None

sync data to remote server

Parameters:

  • path (Path) – local path to sync

  • built_packages (list[Package]) – list of packages which has just been built

ahriman.core.upload.upload module

class Upload(repository_id: RepositoryId, configuration: Configuration)

Bases: LazyLogging

base remote sync class

configuration

configuration instance

Type:

Configuration

repository_id

repository unique identifier

Type:

RepositoryId

Examples

These classes provide the way to upload packages to remote sources as it is described in their implementations. Basic flow includes class instantiating by using the load() method and then calling the run() method which wraps any internal exceptions into the ahriman.core.exceptions.SynchronizationError exception:

>>> configuration = Configuration()
>>> upload = Upload.load(RepositoryId("x86_64", "aur-clone"), configuration, "s3")
>>> upload.run(configuration.repository_paths.repository, [])

Or in case if direct access to exception is required, the sync() method can be used:

>>> try:
>>>     upload.sync(configuration.repository_paths.repository, [])
>>> except Exception as ex:
>>>     handle_exceptions(ex)

default constructor

Parameters:
static load(repository_id: RepositoryId, configuration: Configuration, target: str) → Upload

load client from settings

Parameters:

  • repository_id (RepositoryId) – repository unique identifier

  • configuration (Configuration) – configuration instance

  • target (str) – target to run sync (e.g. s3)

Returns:

client according to current settings

Return type:

Upload

run(path: Path, built_packages: list[Package]) → None

run remote sync

Parameters:

  • path (Path) – local path to sync

  • built_packages (list[Package]) – list of packages which has just been built

Raises:

SynchronizationError – in case of any synchronization unmatched exception

sync(path: Path, built_packages: list[Package]) → None

sync data to remote server

Parameters:

  • path (Path) – local path to sync

  • built_packages (list[Package]) – list of packages which has just been built

ahriman.core.upload.upload_trigger module

class UploadTrigger(repository_id: RepositoryId, configuration: Configuration)

Bases: Trigger

synchronization trigger

targets

upload target list

Type:

list[str]

default constructor

Parameters:
classmethod configuration_sections(configuration: Configuration) → list[str]

extract configuration sections from configuration

Parameters:

configuration (Configuration) – configuration instance

Returns:

read configuration sections belong to this trigger

Return type:

list[str]

on_result(result: Result, packages: list[Package]) → None

run trigger

Parameters:

  • result (Result) – build result

  • packages (list[Package]) – list of all available packages

Module contents