Protocol API ¶

get_toolset()[source]¶

Convert the data present in the class into a dictionary.

Returns: A dictionary with data associated with the class

Return type: Dict

class sail_on_client.protocol.visual_dataclasses.WorldChangeDetectionParams[source]¶

Class for storing parameters associated world change in an algorithm.

__init__(features_dict, logit_dict, round_id, redlight_image)¶

Parameters

features_dict (Dict) –
logit_dict (Dict) –
round_id (int) –
redlight_image (str) –

Return type

get_toolset()[source]¶

Convert the data present in the class into a dictionary.

Returns: A dictionary with data associated with the class

Return type: Dict

Visual Round¶

Source code: sail_on_client/protocol/visual_round.py

Round for visual protocol.

class sail_on_client.protocol.visual_round.VisualRound[source]¶

Class with common elements for visual protocols.

__init__(algorithm, data_root, features_dict, harness, logit_dict, redlight_instance, session_id, skip_stages, test_id)[source]¶

Construct VisualRound.

Parameters

algorithm (Any) – An instance of algorithm
data_root (str) – Root directory of the data
features_dict (Dict) – Dictionary with features for the entire dataset
harness (~TestAndEvaluationHarnessType) – An instance of the harness used for T&E
logit_dict (Dict) – Dictionary with logits for the entire dataset
redlight_instance (str) – The instance when the world changes
session_id (str) – Session id associated with the algorithm
skip_stages (List[str]) – List of stages that are skipped
test_id (str) – Test id associated with the round

Returns

None

Return type

_run_feature_extraction(fe_params, instance_ids)[source]¶

Private helper function for running feature extraction.

Parameters

fe_params (FeatureExtractionParams) – An instance of dataclass with parameters for feature extraction
instance_ids (List[str]) – Identifiers associated with data for a round

Return type

Tuple[Dict, Dict]

Returns

Tuple for feature and logit dictionary for a round

_run_world_change_detection(wcd_params, round_id)[source]¶

Private helper function for detecting that the world has changed.

Parameters

wcd_params (WorldChangeDetectionParams) – An instance of dataclass with parameters for world change detection
round_id (int) – Identifier for a round

Return type

Returns

None

static get_instance_ids(dataset_path)[source]¶

Get instance ids from the dataset.

Parameters: dataset_path (str) – Path to text file with instances used in a round
Return type: List[str]
Returns: List of instance ids from the dataset

Visual Test¶

Source code: sail_on_client/protocol/visual_test.py

Test for Visual Protocol.

class sail_on_client.protocol.visual_test.VisualTest[source]¶

Class representing test in visual protocol.

__init__(algorithm_attributes, data_root, domain, feature_dir, harness, save_dir, session_id, skip_stages, use_consolidated_features, use_saved_features)[source]¶

Construct visual test.

Parameters

algorithm_attributes (Union[AlgorithmAttributes, AlgorithmAttributes]) – An instance of algorithm_attributes
data_root (str) – Root directory for the dataset
domain (str) – Name of the domain for the test
feature_dir (str) – Directory to save features
harness (~TestAndEvaluationHarnessType) – An Instance of harness used for T&E
save_dir (str) – The directory where features are saved
session_id (str) – Session identifier for the test
skip_stages (List[str]) – List of stages that would be skipped
use_consolidated_features (bool) – Flag for using consolidated features
use_saved_features (bool) – Flag for using saved features

Returns

None

Return type

_restore_features(test_id)[source]¶

Private function to restore features.

Parameters: test_id (str) – An identifier for the test
Return type: Tuple[Dict, Dict]
Returns: Tuple of dictionary with features and logits obtained from the feature extractor

_save_features(test_id, feature_dict, logit_dict)[source]¶

Save features for a test.

Parameters

test_id (str) – An identifier for the test
feature_dict (Dict) – Features for the test
logit_dict (Dict) – Logit for the test

Return type

Returns

None

Visual Protocol¶

Source code: sail_on_client/protocol/visual_protocol.py

Visual Protocol.

class sail_on_client.protocol.visual_protocol.VisualProtocol[source]¶

Protocol for visual tasks.

__init__(algorithms, harness)[source]¶

Construct visual protocol.

Parameters

algorithms (Dict[str, ~VisualAgentType]) – Dictionary of algorithms that are used run based on the protocol
harness (~TestAndEvaluationHarnessType) – A harness for test and evaluation

Returns

None

Return type

static __new__(cls, *args, **kwargs)¶

Return type

~P

Parameters

cls (Type[smqtk_core.plugin.P]) –
args (Any) –
kwargs (Any) –

classmethod from_config(config_dict, merge_default=True)[source]¶

Construct protocol from config.

Parameters

config_dict (Dict) – dictionary with parameters
merge_default (bool) – Merge dictionary with default values

Return type

~VisualProtocolType

Returns

An instance of visual protocol

get_config()[source]¶

Get json compliant representation of the protocol.

Return type: Dict
Returns: Dictionary with json compliant representation of the protocol

classmethod get_default_config()¶

Generate and return a default configuration dictionary for this class. This will be primarily used for generating what the configuration dictionary would look like for this class without instantiating it.

By default, we observe what this class’s constructor takes as arguments, turning those argument names into configuration dictionary keys. If any of those arguments have defaults, we will add those values into the configuration dictionary appropriately. The dictionary returned should only contain JSON compliant value types.

It is not be guaranteed that the configuration dictionary returned from this method is valid for construction of an instance of this class.

Returns: Default configuration dictionary for the class.
Return type: dict

>>> # noinspection PyUnresolvedReferences
>>> class SimpleConfig(Configurable):
...     def __init__(self, a=1, b='foo'):
...         self.a = a
...         self.b = b
...     def get_config(self):
...         return {'a': self.a, 'b': self.b}
>>> self = SimpleConfig()
>>> config = self.get_default_config()
>>> assert config == {'a': 1, 'b': 'foo'}

classmethod get_impls()¶

Discover and return a set of classes that implement the calling class.

See the various discover_via_*() functions in this module for more details on the logic of how implementing classes (aka “plugins”) are discovered.

The class-level variables PLUGIN_ENV_VAR and PLUGIN_NAMESPACE may be overridden to change what environment and entry-point extension are looked for, respectively.

Return type: Set[Type[~P]]
Returns: Set of discovered class types that are considered “valid” plugins of this type. See is_valid_plugin() for what we define a “valid” type to be be relative to this class.

classmethod is_usable()¶

Determine if this class will be detected by SMQTK’s plugin utilities.

Return type: bool

abstract run_protocol(config)¶

Run the protocol.

Return type: None
Parameters: config (Dict[str, Any]) –

OND API¶

OND Dataclasses¶

Source code: sail_on_client/protocol/ond_dataclasses.py

Dataclasses for the protocols.

class sail_on_client.protocol.ond_dataclasses.AlgorithmAttributes[source]¶

Class for storing attributes of algorithm present in the protocol.

__init__(name, detection_threshold, instance, is_baseline, is_reaction_baseline, package_name, parameters, session_id, test_ids)¶

Parameters

name (str) –
detection_threshold (float) –
instance (Any) –
is_baseline (bool) –
is_reaction_baseline (bool) –
package_name (str) –
parameters (Dict) –
session_id (str) –
test_ids (List[str]) –

Return type

merge_detector_params(detector_params, exclude_keys=None)[source]¶

Merge common parameters with algorithm specific parameters with exclusions.

Parameters

detector_params (Dict) – Dictionary of common parameters
exclude_keys (Optional[List]) – List of keys that should be excluded in the merge

Return type

Returns

None

named_version()[source]¶

Compute version of an algorithm.

Return type: str
Returns: A string containing name and the version number

remove_completed_tests(finished_tests)[source]¶

Remove finished tests from test_ids.

Parameters: finished_tests (List[str]) – List of tests that are complete
Return type: None
Returns: None

class sail_on_client.protocol.ond_dataclasses.InitializeParams[source]¶

Class for storing parameters that are used to initialize the algorithm.

__init__(parameters, session_id, test_id, pre_novelty_batches, feedback_instance)¶

Parameters

parameters (Dict) –
session_id (str) –
test_id (str) –
pre_novelty_batches (int) –
feedback_instance (Optional[Union[sail_on_client.feedback.image_classification_feedback.ImageClassificationFeedback, sail_on_client.feedback.document_transcription_feedback.DocumentTranscriptionFeedback, sail_on_client.feedback.activity_recognition_feedback.ActivityRecognitionFeedback]]) –

Return type

get_toolset()[source]¶

Convert the data present in the class into a dictionary.

Returns: A dictionary with data associated with the class

Return type: Dict

class sail_on_client.protocol.ond_dataclasses.NoveltyAdaptationParams[source]¶

Class for storing parameters associated novelty adaptation with an algorithm.

__init__(round_id)¶

Parameters: round_id (int) –
Return type: None

get_toolset()[source]¶

Convert the data present in the class into a dictionary.

Returns: A dictionary with data associated with the class

Return type: Dict

class sail_on_client.protocol.ond_dataclasses.NoveltyCharacterizationParams[source]¶

Class for storing parameters associated novelty characterization with an algorithm.

__init__(dataset_ids)¶

Parameters: dataset_ids (List[str]) –
Return type: None

get_toolset()[source]¶

Convert the data present in the class into a dictionary.

Returns: A dictionary with data associated with the class

Return type: Dict

class sail_on_client.protocol.ond_dataclasses.NoveltyClassificationParams[source]¶

Class for storing parameters associated novelty classification in an algorithm.

__init__(features_dict, logit_dict, round_id)¶

Parameters

features_dict (Dict) –
logit_dict (Dict) –
round_id (int) –

Return type

get_toolset()[source]¶

Convert the data present in the class into a dictionary.

Returns: A dictionary with data associated with the class

Return type: Dict

OND Round¶

Source code: sail_on_client/protocol/ond_round.py

Round for OND.

class sail_on_client.protocol.ond_round.ONDRound[source]¶

Class Representing a round in OND.

__call__(dataset, round_id)[source]¶

Core logic for running round in OND.

Parameters

algorithm – An instance of the algorithm
dataset (str) – Path to a file with the dataset for the round
round_id (int) – An Identifier for a round

Return type

Optional[Dict]

Returns

Score for the round

__init__(algorithm, data_root, features_dict, harness, logit_dict, redlight_instance, session_id, skip_stages, test_id)[source]¶

Construct round for OND.

Parameters

algorithm (Any) – An instance of algorithm
data_root (str) – Root directory of the data
features_dict (Dict) – Dictionary with features for the entire dataset
harness (~TestAndEvaluationHarnessType) – An instance of the harness used for T&E
logit_dict (Dict) – Dictionary with logits for the entire dataset
redlight_instance (str) – The instance when the world changes
session_id (str) – Session id associated with the algorithm
skip_stages (List[str]) – List of stages that are skipped
test_id (str) – Test id associated with the round

Returns

None

Return type

_run_novelty_adaptation(na_params)[source]¶

Private helper function for adaptation.

Parameters: na_params (NoveltyAdaptationParams) – An instance of dataclass with parameters for adaptation
Return type: None
Returns: None

_run_novelty_classification(nc_params, round_id)[source]¶

Private helper function for novelty classification.

Parameters

nc_params (NoveltyClassificationParams) – An instance of dataclass with parameters for novelty classification
round_id (int) – Identifier for a round

Return type

Returns

None

static get_instance_ids(dataset_path)¶

Get instance ids from the dataset.

Parameters: dataset_path (str) – Path to text file with instances used in a round
Return type: List[str]
Returns: List of instance ids from the dataset

OND Test¶

Source code: sail_on_client/protocol/ond_test.py

Test for OND.

class sail_on_client.protocol.ond_test.ONDTest[source]¶

Class Representing OND test.

__call__(test_id)[source]¶

Core logic for running test in OND.

Parameters: test_id (str) – An identifier for the test
Return type: Optional[Dict]
Returns: Score for the test

__init__(algorithm_attributes, data_root, domain, feedback_type, feature_dir, harness, save_dir, session_id, skip_stages, use_consolidated_features, use_saved_features)[source]¶

Construct test for OND.

Parameters

algorithm_attributes (AlgorithmAttributes) – An instance of algorithm_attributes
data_root (str) – Root directory for the dataset
domain (str) – Name of the domain for the test
feedback_type (str) – Type of feedback used in the test
harness (~TestAndEvaluationHarnessType) – An Instance of harness used for T&E
save_dir (str) – The directory where features are saved
session_id (str) – Session identifier for the test
skip_stages (List[str]) – List of stages that would be skipped
use_consolidated_features (bool) – Flag for using consolidated features
use_saved_features (bool) – Flag for using saved features
feature_dir (str) –

Returns

None

Return type

Union[ActivityRecognitionFeedback, DocumentTranscriptionFeedback, ImageClassificationFeedback]

_create_feedback_instance(test_id, feedback_max_ids)[source]¶

Private function for creating feedback object.

Parameters

test_id (str) – An identifier for the test
feedback_max_ids (int) – Budget provided in metadata

Return type

Returns

An instance of feedback for the domain

OND Protocol¶

Source code: sail_on_client/protocol/ond_protocol.py

OND protocol.

class sail_on_client.protocol.ond_protocol.ONDProtocol[source]¶

OND protocol.

__init__(algorithms, dataset_root, domain, harness, save_dir, seed, test_ids, baseline_class='', feature_extraction_only=False, has_baseline=False, has_reaction_baseline=False, hints=None, is_eval_enabled=False, is_eval_roundwise_enabled=False, resume_session=False, resume_session_ids=None, save_attributes=False, saved_attributes=None, save_elementwise=False, save_features=False, feature_dir='', skip_stages=None, use_feedback=False, feedback_type='classification', use_consolidated_features=False, use_saved_attributes=False, use_saved_features=False)[source]¶

Construct OND protocol.

Parameters

algorithms (Dict[str, ONDAgent]) – Dictionary of algorithms that are used run based on the protocol
baseline_class (str) – Name of the baseline class
dataset_root (str) – Root directory of the dataset
domain (str) – Domain of the problem
save_dir (str) – Directory where results are saved
seed (str) – Seed for the experiments
feedback_type (str) – Type of feedback
test_ids (List[str]) – List of tests
feature_extraction_only (bool) – Flag to only run feature extraction
has_baseline (bool) – Flag to check if the session has baseline
has_reaction_baseline (bool) – Flag to check if the session has reaction baseline
hints (Optional[List]) – List of hint provided in the session
harness (~TestAndEvaluationHarnessType) – A harness for test and evaluation
is_eval_enabled (bool) – Flag to check if evaluation is enabled in session
is_eval_roundwise_enabled (bool) – Flag to check if evaluation is enabled for rounds
resume_session (bool) – Flag to resume session
resume_session_ids (Optional[Dict]) – Dictionary for resuming sessions
save_attributes (bool) – Flag to save attributes
saved_attributes (Optional[Dict]) – Dictionary for attributes
save_elementwise (bool) – Flag to save features elementwise
save_features (bool) – Flag to save features
feature_dir (str) – Directory to save features
skip_stages (Optional[List]) – List of stages that are skipped
use_feedback (bool) – Flag to use feedback
use_saved_attributes (bool) – Flag to use saved attributes
use_saved_features (bool) – Flag to use saved features
use_consolidated_features (bool) –

Returns

None

Return type

static __new__(cls, *args, **kwargs)¶

Return type

~P

Parameters

cls (Type[smqtk_core.plugin.P]) –
args (Any) –
kwargs (Any) –

_evaluate_algorithms(algorithms_attributes, algorithm_scores, save_dir)[source]¶

Evaluate algorithms after all tests have been submitted.

Parameters

algorithms_attributes (List[AlgorithmAttributes]) – All algorithms present in the config
algorithm_scores (Dict) – Scores for round wise evaluation
save_dir (str) – Directory where the scores are stored

Return type

Returns

None

create_algorithm_attributes(algorithm_name, algorithm_param, baseline_algorithm_name, has_baseline, has_reaction_baseline, test_ids)[source]¶

Create an instance of algorithm attributes.

Parameters

algorithm_name (str) – Name of the algorithm
algorithm_param (Dict) – Parameters for the algorithm
baseline_algorithm_name (str) – Name of the baseline algorithm
has_baseline (bool) – Flag to check if a baseline is present in the config
has_reaction_baseline (bool) – Flag to check if a reaction baseline is present in the config
test_ids (List[str]) – List of test

Return type

Returns

An instance of AlgorithmAttributes

create_algorithm_session(algorithm_attributes, domain, hints, has_a_session, protocol_name)[source]¶

Create/resume session for an algorithm.

Parameters

algorithm_attributes (AlgorithmAttributes) – An instance of AlgorithmAttributes
domain (str) – Domain for the algorithm
hints (List[str]) – List of hints used in the session
has_a_session (bool) – Already has a session and we want to resume it
protocol_name (str) – Name of the algorithm

Return type

Returns

An AlgorithmAttributes object with updated session id or test id

classmethod from_config(config_dict, merge_default=True)¶

Construct protocol from config.

Parameters

config_dict (Dict) – dictionary with parameters
merge_default (bool) – Merge dictionary with default values

Return type

~VisualProtocolType

Returns

An instance of visual protocol

get_config()[source]¶

Get dictionary representation of the object.

Return type: Dict

classmethod get_default_config()¶

Generate and return a default configuration dictionary for this class. This will be primarily used for generating what the configuration dictionary would look like for this class without instantiating it.

By default, we observe what this class’s constructor takes as arguments, turning those argument names into configuration dictionary keys. If any of those arguments have defaults, we will add those values into the configuration dictionary appropriately. The dictionary returned should only contain JSON compliant value types.

It is not be guaranteed that the configuration dictionary returned from this method is valid for construction of an instance of this class.

Returns: Default configuration dictionary for the class.
Return type: dict

>>> # noinspection PyUnresolvedReferences
>>> class SimpleConfig(Configurable):
...     def __init__(self, a=1, b='foo'):
...         self.a = a
...         self.b = b
...     def get_config(self):
...         return {'a': self.a, 'b': self.b}
>>> self = SimpleConfig()
>>> config = self.get_default_config()
>>> assert config == {'a': 1, 'b': 'foo'}

classmethod get_impls()¶

Discover and return a set of classes that implement the calling class.

See the various discover_via_*() functions in this module for more details on the logic of how implementing classes (aka “plugins”) are discovered.

The class-level variables PLUGIN_ENV_VAR and PLUGIN_NAMESPACE may be overridden to change what environment and entry-point extension are looked for, respectively.

Return type: Set[Type[~P]]
Returns: Set of discovered class types that are considered “valid” plugins of this type. See is_valid_plugin() for what we define a “valid” type to be be relative to this class.

classmethod is_usable()¶

Determine if this class will be detected by SMQTK’s plugin utilities.

Return type: bool

run_protocol(config)[source]¶

Run the protocol.

Parameters: config (Dict) – Parameters provided in the config
Return type: None
Returns: None

update_skip_stages(skip_stages, is_eval_enabled, is_eval_roundwise_enabled, use_feedback, save_features, feature_extraction_only)[source]¶

Update skip stages based on the boolean values in config.

Parameters

skip_stages (List[str]) – List of skip stages specified in the config
is_eval_enabled (bool) – Flag to enable evaluation
is_eval_roundwise_enabled (bool) – Flag to enable evaluation in every round
use_feedback (bool) – Flag to enable using feedback
save_features (bool) – Flag to enable saving features
feature_extraction_only (bool) – Flag to only run feature extraction

Return type

List[str]

Returns

Update list of skip stages

CONDDA API¶

CONDDA Dataclasses¶

Source code: sail_on_client/protocol/condda_dataclasses.py

Dataclasses for CONDDA.

class sail_on_client.protocol.condda_dataclasses.AlgorithmAttributes[source]¶

Class for storing attributes of algorithm present in the protocol.

__init__(name, detection_threshold, instance, package_name, parameters, session_id, test_ids)¶

Parameters

name (str) –
detection_threshold (float) –
instance (Any) –
package_name (str) –
parameters (Dict) –
session_id (str) –
test_ids (List[str]) –

Return type

merge_detector_params(detector_params, exclude_keys=None)[source]¶

Merge common parameters with algorithm specific parameters with exclusions.

Parameters

detector_params (Dict) – Dictionary of common parameters
exclude_keys (Optional[List]) – List of keys that should be excluded in the merge

Return type

Returns

None

named_version()[source]¶

Compute version of an algorithm.

Return type: str
Returns: A string containing name and the version number

remove_completed_tests(finished_tests)[source]¶

Remove finished tests from test_ids.

Parameters: finished_tests (List[str]) – List of tests that are complete
Return type: None
Returns: None

class sail_on_client.protocol.condda_dataclasses.InitializeParams[source]¶

Class for storing parameters that are used to initialize the algorithm.

__init__(parameters, session_id, test_id)¶

Parameters

parameters (Dict) –
session_id (str) –
test_id (str) –

Return type

get_toolset()[source]¶

Convert the data present in the class into a dictionary.

Returns: A dictionary with data associated with the class

Return type: Dict

class sail_on_client.protocol.condda_dataclasses.NoveltyCharacterizationParams[source]¶

Class for storing parameters associated novelty characterization with an algorithm.

__init__(features_dict, logit_dict, round_id)¶

Parameters

features_dict (Dict) –
logit_dict (Dict) –
round_id (int) –

Return type

get_toolset()[source]¶

Convert the data present in the class into a dictionary.

Returns: A dictionary with data associated with the class

Return type: Dict

CONDDA Round¶

Source code: sail_on_client/protocol/condda_round.py

Round for CONDDA.

class sail_on_client.protocol.condda_round.CONDDARound[source]¶

Class Representing a round for CONDDA.

__call__(dataset, round_id)[source]¶

Core logic for running round in CONDDA.

Parameters

dataset (str) – Path to a file with the dataset for the round
round_id (int) – An Identifier for a round

Return type

Returns

None

__init__(algorithm, data_root, features_dict, harness, logit_dict, redlight_instance, session_id, skip_stages, test_id)[source]¶

Construct CONDDARound.

Parameters

algorithm (Any) – An instance of algorithm
data_root (str) – Root directory of the data
features_dict (Dict) – Dictionary with features for the entire dataset
harness (~TestAndEvaluationHarnessType) – An instance of the harness used for T&E
logit_dict (Dict) – Dictionary with logits for the entire dataset
redlight_instance (str) – The instance when the world changes
session_id (str) – Session id associated with the algorithm
skip_stages (List[str]) – List of stages that are skipped
test_id (str) – Test id associated with the round

Returns

None

Return type

static get_instance_ids(dataset_path)¶

Get instance ids from the dataset.

Parameters: dataset_path (str) – Path to text file with instances used in a round
Return type: List[str]
Returns: List of instance ids from the dataset

CONDDA Test¶

Source code: sail_on_client/protocol/condda_test.py

Test for CONDDA.

class sail_on_client.protocol.condda_test.CONDDATest[source]¶

Class representing CONDDA Test.

__call__(test_id)[source]¶

Core logic for running test in CONDDA.

Parameters: test_id (str) – An identifier for the test
Return type: None
Returns: Score for the test

__init__(algorithm_attributes, data_root, domain, feature_dir, harness, save_dir, session_id, skip_stages, use_consolidated_features, use_saved_features)[source]¶

Construct test for CONDDA.

Parameters

algorithm_attributes (AlgorithmAttributes) – An instance of algorithm_attributes
data_root (str) – Root directory for the dataset
domain (str) – Name of the domain for the test
feature_dir (str) – Directory where features are stored
harness (~TestAndEvaluationHarnessType) – An Instance of harness used for T&E
save_dir (str) – The directory where features are saved
session_id (str) – Session identifier for the test
skip_stages (List[str]) – List of stages that would be skipped
use_consolidated_features (bool) – Flag for using consolidated features
use_saved_features (bool) – Flag for using saved features

Returns

None

Return type

CONDDA Protocol¶

Source code: sail_on_client/protocol/condda_protocol.py

CONDDA protocol.

class sail_on_client.protocol.condda_protocol.Condda[source]¶

CONDDA protocol.

__init__(algorithms, dataset_root, domain, harness, save_dir, seed, test_ids, feature_extraction_only=False, hints=None, is_eval_enabled=False, is_eval_roundwise_enabled=False, resume_session=False, resume_session_ids=None, save_attributes=False, saved_attributes=None, save_elementwise=False, save_features=False, feature_dir='', skip_stages=None, use_consolidated_features=False, use_saved_attributes=False, use_saved_features=False)[source]¶

Initialize CONDDA protocol object.

Parameters

algorithms (Dict[str, CONDDAAgent]) – Dictionary of algorithms that are used run based on the protocol
dataset_root (str) – Root directory of the dataset
domain (str) – Domain of the problem
save_dir (str) – Directory where results are saved
seed (str) – Seed for the experiments
test_ids (List[str]) – List of tests
feature_extraction_only (bool) – Flag to only run feature extraction
hints (Optional[List]) – List of hint provided in the session
harness (~TestAndEvaluationHarnessType) – A harness for test and evaluation
is_eval_enabled (bool) – Flag to check if evaluation is enabled in session
is_eval_roundwise_enabled (bool) – Flag to check if evaluation is enabled for rounds
resume_session (bool) – Flag to resume session
resume_session_ids (Optional[Dict]) – Dictionary for resuming sessions
save_attributes (bool) – Flag to save attributes
saved_attributes (Optional[Dict]) – Dictionary for attributes
save_elementwise (bool) – Flag to save features elementwise
save_features (bool) – Flag to save features
feature_dir (str) – Directory to save features
skip_stages (Optional[List]) – List of stages that are skipped
use_consolidated_features (bool) – Flag to use consolidated features
use_saved_attributes (bool) – Flag to use saved attributes
use_saved_features (bool) – Flag to use saved features

Returns

None

Return type

static __new__(cls, *args, **kwargs)¶

Return type

~P

Parameters

cls (Type[smqtk_core.plugin.P]) –
args (Any) –
kwargs (Any) –

create_algorithm_attributes(algorithm_name, algorithm_param, test_ids)[source]¶

Create an instance of algorithm attributes.

Parameters

algorithm_name (str) – Name of the algorithm
algorithm_param (Dict) – Parameters for the algorithm
baseline_algorithm_name – Name of the baseline algorithm
has_baseline – Flag to check if a baseline is present in the config
has_reaction_baseline – Flag to check if a reaction baseline is present in the config
test_ids (List[str]) – List of test

Return type

Returns

An instance of AlgorithmAttributes

create_algorithm_session(algorithm_attributes, domain, hints, has_a_session, protocol_name)[source]¶

Create/resume session for an algorithm.

Parameters

algorithm_attributes (AlgorithmAttributes) – An instance of AlgorithmAttributes
domain (str) – Domain for the algorithm
hints (List[str]) – List of hints used in the session
has_a_session (bool) – Already has a session and we want to resume it
protocol_name (str) – Name of the algorithm

Return type