Harness API ¶

Returns

None

abstract dataset_request(test_id, round_id, session_id)[source]¶

Request data for evaluation.

Parameters

test_id (str) – The test being evaluated at this moment.
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The identifier provided by the server for a single experiment

Return type

Returns

Filename of a file containing a list of image files (including full path for each)

abstract evaluate(test_id, round_id, session_id, baseline_session_id=None)[source]¶

Get results for test(s).

Parameters

test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session
baseline_session_id (Optional[str]) –

Return type

Dict

Returns

Path to a file with the results

abstract evaluate_round_wise(test_id, round_id, session_id)[source]¶

Get results for round(s).

Parameters

test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session

Return type

Returns

Path to a file with the results

classmethod from_config(config_dict, merge_default=True)¶

Instantiate a new instance of this class given the configuration JSON-compliant dictionary encapsulating initialization arguments.

This base method is adequate without modification when a class’s constructor argument types are JSON-compliant. If one or more are not, however, this method then needs to be overridden in order to convert from a JSON-compliant stand-in into the more complex object the constructor requires. It is recommended that when complex types are used they also inherit from the Configurable in order to hopefully make easier the conversion to and from JSON-compliant stand-ins.

When this method does need to be overridden, this usually looks like the following pattern:

D = TypeVar("D", bound="MyClass")

class MyClass (Configurable):

    @classmethod
    def from_config(
        cls: Type[D],
        config_dict: Dict,
        merge_default: bool = True
    ) -> D:
        # Perform a shallow copy of the input ``config_dict`` which
        # is important to maintain idempotency.
        config_dict = dict(config_dict)

        # Optionally guarantee default values are present in the
        # configuration dictionary.  This is useful when the
        # configuration dictionary input is partial and the logic
        # contained here wants to use config parameters that may
        # have defaults defined in the constructor.
        if merge_default:
            config_dict = merge_dict(cls.get_default_config(),
                                     config_dict)

        #
        # Perform any overriding of `config_dict` values here.
        #

        # Create and return an instance using the super method.
        return super().from_config(config_dict,
                                   merge_default=merge_default)

Note on type annotations: When defining a sub-class of configurable and override this class method, we will need to defined a new TypeVar that is bound at the new class type. This is because super requires a type to be given that descends from the implementing type. If C is used as defined in this interface module, which is upper-bounded on the base Configurable class, the type analysis will see that we are attempting to invoke super with a type that may not strictly descend from the implementing type (MyClass in the example above), and cause an error during type analysis.

Parameters

config_dict (dict) – JSON compliant dictionary encapsulating a configuration.
merge_default (bool) – Merge the given configuration on top of the default provided by get_default_config.

Return type

~C

Returns

Constructed instance from the provided config.

get_config()[source]¶

Return a default configuration dictionary.

Return type: Dict

classmethod get_default_config()¶

Generate and return a default configuration dictionary for this class. This will be primarily used for generating what the configuration dictionary would look like for this class without instantiating it.

By default, we observe what this class’s constructor takes as arguments, turning those argument names into configuration dictionary keys. If any of those arguments have defaults, we will add those values into the configuration dictionary appropriately. The dictionary returned should only contain JSON compliant value types.

It is not be guaranteed that the configuration dictionary returned from this method is valid for construction of an instance of this class.

Returns: Default configuration dictionary for the class.
Return type: dict

>>> # noinspection PyUnresolvedReferences
>>> class SimpleConfig(Configurable):
...     def __init__(self, a=1, b='foo'):
...         self.a = a
...         self.b = b
...     def get_config(self):
...         return {'a': self.a, 'b': self.b}
>>> self = SimpleConfig()
>>> config = self.get_default_config()
>>> assert config == {'a': 1, 'b': 'foo'}

abstract get_feedback_request(feedback_ids, feedback_type, test_id, round_id, session_id)[source]¶

Get Labels from the server based provided one or more example ids.

Parameters

feedback_ids (list) – List of media ids for which feedback is required
feedback_type (str) – Protocols constants with the values: label, detection, characterization
test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session

Return type

Returns

Path to a file containing containing requested feedback

classmethod get_impls()¶

Discover and return a set of classes that implement the calling class.

See the various discover_via_*() functions in this module for more details on the logic of how implementing classes (aka “plugins”) are discovered.

The class-level variables PLUGIN_ENV_VAR and PLUGIN_NAMESPACE may be overridden to change what environment and entry-point extension are looked for, respectively.

Return type: Set[Type[~P]]
Returns: Set of discovered class types that are considered “valid” plugins of this type. See is_valid_plugin() for what we define a “valid” type to be be relative to this class.

abstract get_test_metadata(session_id, test_id)[source]¶

Retrieve the metadata json for the specified test.

Parameters

session_id (str) – The id of the session currently being evaluated
test_id (str) – The id of the test currently being evaluated

Return type

Returns

A dictionary containing metadata

classmethod is_usable()[source]¶

Determine if this class with be detected by SMQTK’s plugin.

Return type: bool

abstract post_results(result_files, test_id, round_id, session_id)[source]¶

Post client detector predictions for the dataset.

Parameters

result_files (Dict[str, str]) – A dictionary of results with protocol constant as key and file path as value
test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session

Return type

Returns

None

abstract resume_session(session_id)[source]¶

Get finished test from an existing session.

Parameters

id (session) – Session id that was started but not terminated
session_id (str) –

Return type

List[str]

Returns

List of tests finished in the session

abstract session_request(test_ids, protocol, domain, novelty_detector_version, hints, detection_threshold)[source]¶

Create a new session to evaluate the detector using an empirical protocol.

Parameters

test_ids (list) – List of tests being evaluated in this session
protocol (str) – String indicating which protocol is being evaluated
domain (str) – String indicating which domain is being evaluated
novelty_detector_version (str) – The novelty detector being evaluated
hints (list) – Hints used for the session
detection_threshold (float) – Detection threshold for the session

Return type

Returns

A session identifier provided by the server

abstract terminate_session(session_id)[source]¶

Terminate the session after the evaluation for the protocol is complete.

Parameters: session_id (str) – The id provided by a server denoting a session
Return type: None

Returns: None

Return type: None
Parameters: session_id (str) –

abstract test_ids_request(protocol, domain, detector_seed, test_assumptions='{}')[source]¶

Request Test Identifiers as part of a series of individual tests.

Parameters

protocol (str) – string indicating which protocol is being evaluated
domain (str) – problem domain for the tests
detector_seed (str) – A seed provided by the novelty detector
test_assumptions (str) – Assumptions used by the detector

Return type

Returns

Filename of file containing test ids

Local Harness¶

Source code: sail_on_client/harness/local_harness.py

Implementation of T&E Harness for running experiments locally.

class sail_on_client.harness.local_harness.LocalHarness[source]¶

Harness without any server communication.

__init__(data_dir, result_dir, gt_dir='', gt_config='')[source]¶

Initialize an object of local harness.

Parameters

data_dir (str) – Path to the directory with the data
result_dir (str) – Path to the directory where results are stored
gt_dir (str) – Path to directory with ground truth
gt_config (str) – Path to config file with column mapping for ground truth

Returns

None

Return type

None

static __new__(cls, *args, **kwargs)¶

Return type

~P

Parameters

cls (Type[smqtk_core.plugin.P]) –
args (Any) –
kwargs (Any) –

complete_test(session_id, test_id)[source]¶

Mark test as completed.

Parameters

session_id (str) – The id of the session currently being evaluated
test_id (str) – The id of the test currently being evaluated

Return type

Returns

None

dataset_request(test_id, round_id, session_id)[source]¶

Request data for evaluation.

Parameters

test_id (str) – The test being evaluated at this moment.
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The identifier provided by the server for a single experiment

Return type

Returns

Filename of a file containing a list of image files (including full path for each)

evaluate(test_id, round_id, session_id, baseline_session_id=None)[source]¶

Get results for test(s).

Parameters

test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session
baseline_session_id (Optional[str]) –

Return type

Returns

Path to a file with the results

evaluate_round_wise(test_id, round_id, session_id)[source]¶

Get results for round(s).

Parameters

test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session

Return type

Returns

Path to a file with the results

classmethod from_config(config_dict, merge_default=True)¶

Instantiate a new instance of this class given the configuration JSON-compliant dictionary encapsulating initialization arguments.

This base method is adequate without modification when a class’s constructor argument types are JSON-compliant. If one or more are not, however, this method then needs to be overridden in order to convert from a JSON-compliant stand-in into the more complex object the constructor requires. It is recommended that when complex types are used they also inherit from the Configurable in order to hopefully make easier the conversion to and from JSON-compliant stand-ins.

When this method does need to be overridden, this usually looks like the following pattern:

D = TypeVar("D", bound="MyClass")

class MyClass (Configurable):

    @classmethod
    def from_config(
        cls: Type[D],
        config_dict: Dict,
        merge_default: bool = True
    ) -> D:
        # Perform a shallow copy of the input ``config_dict`` which
        # is important to maintain idempotency.
        config_dict = dict(config_dict)

        # Optionally guarantee default values are present in the
        # configuration dictionary.  This is useful when the
        # configuration dictionary input is partial and the logic
        # contained here wants to use config parameters that may
        # have defaults defined in the constructor.
        if merge_default:
            config_dict = merge_dict(cls.get_default_config(),
                                     config_dict)

        #
        # Perform any overriding of `config_dict` values here.
        #

        # Create and return an instance using the super method.
        return super().from_config(config_dict,
                                   merge_default=merge_default)

Note on type annotations: When defining a sub-class of configurable and override this class method, we will need to defined a new TypeVar that is bound at the new class type. This is because super requires a type to be given that descends from the implementing type. If C is used as defined in this interface module, which is upper-bounded on the base Configurable class, the type analysis will see that we are attempting to invoke super with a type that may not strictly descend from the implementing type (MyClass in the example above), and cause an error during type analysis.

Parameters

config_dict (dict) – JSON compliant dictionary encapsulating a configuration.
merge_default (bool) – Merge the given configuration on top of the default provided by get_default_config.

Return type

~C

Returns

Constructed instance from the provided config.

get_config()[source]¶

JSON Compliant representation of the object.

Return type: Dict

classmethod get_default_config()¶

Generate and return a default configuration dictionary for this class. This will be primarily used for generating what the configuration dictionary would look like for this class without instantiating it.

By default, we observe what this class’s constructor takes as arguments, turning those argument names into configuration dictionary keys. If any of those arguments have defaults, we will add those values into the configuration dictionary appropriately. The dictionary returned should only contain JSON compliant value types.

It is not be guaranteed that the configuration dictionary returned from this method is valid for construction of an instance of this class.

Returns: Default configuration dictionary for the class.
Return type: dict

>>> # noinspection PyUnresolvedReferences
>>> class SimpleConfig(Configurable):
...     def __init__(self, a=1, b='foo'):
...         self.a = a
...         self.b = b
...     def get_config(self):
...         return {'a': self.a, 'b': self.b}
>>> self = SimpleConfig()
>>> config = self.get_default_config()
>>> assert config == {'a': 1, 'b': 'foo'}

get_feedback_request(feedback_ids, feedback_type, test_id, round_id, session_id)[source]¶

Get Labels from the server based provided one or more example ids.

Parameters

feedback_ids (list) – List of media ids for which feedback is required
feedback_type (str) – Protocols constants with the values: label, detection, characterization
test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session

Return type

Returns

Path to a file containing containing requested feedback

classmethod get_impls()¶

Discover and return a set of classes that implement the calling class.

See the various discover_via_*() functions in this module for more details on the logic of how implementing classes (aka “plugins”) are discovered.

The class-level variables PLUGIN_ENV_VAR and PLUGIN_NAMESPACE may be overridden to change what environment and entry-point extension are looked for, respectively.

Return type: Set[Type[~P]]
Returns: Set of discovered class types that are considered “valid” plugins of this type. See is_valid_plugin() for what we define a “valid” type to be be relative to this class.

get_test_metadata(session_id, test_id)[source]¶

Retrieve the metadata json for the specified test.

Parameters

session_id (str) – The id of the session currently being evaluated
test_id (str) – The id of the test currently being evaluated

Return type

Returns

A dictionary containing metadata

classmethod is_usable()¶

Determine if this class with be detected by SMQTK’s plugin.

Return type: bool

post_results(result_files, test_id, round_id, session_id)[source]¶

Post client detector predictions for the dataset.

Parameters

result_files (Dict[str, str]) – A dictionary of results with protocol constant as key and file path as value
test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session

Return type

Returns

None

resume_session(session_id)[source]¶

Get finished test from an existing session.

Parameters

id (session) – Session id that was started but not terminated
session_id (str) –

Return type

List[str]

Returns

List of tests finished in the session

session_request(test_ids, protocol, domain, novelty_detector_version, hints, detection_threshold)[source]¶

Create a new session to evaluate the detector using an empirical protocol.

Parameters

test_ids (list) – List of tests being evaluated in this session
protocol (str) – String indicating which protocol is being evaluated
domain (str) – String indicating which domain is being evaluated
novelty_detector_version (str) – The novelty detector being evaluated
hints (list) – Hints used for the session
detection_threshold (float) – Detection threshold for the session

Return type

Returns

A session identifier provided by the server

terminate_session(session_id)[source]¶

Terminate the session after the evaluation for the protocol is complete.

Parameters: session_id (str) – The id provided by a server denoting a session
Return type: None

Returns: None

Return type: None
Parameters: session_id (str) –

test_ids_request(protocol, domain, detector_seed, test_assumptions='{}')[source]¶

Request Test Identifiers as part of a series of individual tests.

Parameters

protocol (str) – string indicating which protocol is being evaluated
domain (str) – problem domain for the tests
detector_seed (str) – A seed provided by the novelty detector
test_assumptions (str) – Assumptions used by the detector

Return type

Returns

Filename of file containing test ids

PAR Harness¶

Source code: sail_on_client/harness/par_harness.py

Implementation of T&E Harness for PAR Server.

class sail_on_client.harness.par_harness.ParHarness[source]¶

Harness for PAR server.

__init__(url, save_directory)[source]¶

Initialize a client connection object.

Parameters

url (str) – URL for the server
save_directory (str) – A directory to save files

Returns

None

Return type

None

static __new__(cls, *args, **kwargs)¶

Return type

~P

Parameters

cls (Type[smqtk_core.plugin.P]) –
args (Any) –
kwargs (Any) –

complete_test(session_id, test_id)[source]¶

Mark test as completed.

Parameters

session_id (str) – The id of the session currently being evaluated
test_id (str) – The id of the test currently being evaluated

Return type

Returns

None

dataset_request(test_id, round_id, session_id)[source]¶

Request data for evaluation.

Parameters

test_id (str) – The test being evaluated at this moment.
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The identifier provided by the server for a single experiment

Return type

Returns

Filename of a file containing a list of image files (including full path for each)

evaluate(test_id, round_id, session_id, baseline_session_id=None)[source]¶

Get results for test(s).

Parameters

test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session
baseline_session_id (str) –

Return type

Dict

Returns

Path to a file with the results

evaluate_round_wise(test_id, round_id, session_id)[source]¶

Get results for round(s).

Parameters

test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session

Return type

Returns

Path to a file with the results

classmethod from_config(config_dict, merge_default=True)¶

Instantiate a new instance of this class given the configuration JSON-compliant dictionary encapsulating initialization arguments.

This base method is adequate without modification when a class’s constructor argument types are JSON-compliant. If one or more are not, however, this method then needs to be overridden in order to convert from a JSON-compliant stand-in into the more complex object the constructor requires. It is recommended that when complex types are used they also inherit from the Configurable in order to hopefully make easier the conversion to and from JSON-compliant stand-ins.

When this method does need to be overridden, this usually looks like the following pattern:

D = TypeVar("D", bound="MyClass")

class MyClass (Configurable):

    @classmethod
    def from_config(
        cls: Type[D],
        config_dict: Dict,
        merge_default: bool = True
    ) -> D:
        # Perform a shallow copy of the input ``config_dict`` which
        # is important to maintain idempotency.
        config_dict = dict(config_dict)

        # Optionally guarantee default values are present in the
        # configuration dictionary.  This is useful when the
        # configuration dictionary input is partial and the logic
        # contained here wants to use config parameters that may
        # have defaults defined in the constructor.
        if merge_default:
            config_dict = merge_dict(cls.get_default_config(),
                                     config_dict)

        #
        # Perform any overriding of `config_dict` values here.
        #

        # Create and return an instance using the super method.
        return super().from_config(config_dict,
                                   merge_default=merge_default)

Note on type annotations: When defining a sub-class of configurable and override this class method, we will need to defined a new TypeVar that is bound at the new class type. This is because super requires a type to be given that descends from the implementing type. If C is used as defined in this interface module, which is upper-bounded on the base Configurable class, the type analysis will see that we are attempting to invoke super with a type that may not strictly descend from the implementing type (MyClass in the example above), and cause an error during type analysis.

Parameters

config_dict (dict) – JSON compliant dictionary encapsulating a configuration.
merge_default (bool) – Merge the given configuration on top of the default provided by get_default_config.

Return type

~C

Returns

Constructed instance from the provided config.

get_config()[source]¶

JSON Compliant representation of the object.

Return type: Dict

classmethod get_default_config()¶

Generate and return a default configuration dictionary for this class. This will be primarily used for generating what the configuration dictionary would look like for this class without instantiating it.

By default, we observe what this class’s constructor takes as arguments, turning those argument names into configuration dictionary keys. If any of those arguments have defaults, we will add those values into the configuration dictionary appropriately. The dictionary returned should only contain JSON compliant value types.

It is not be guaranteed that the configuration dictionary returned from this method is valid for construction of an instance of this class.

Returns: Default configuration dictionary for the class.
Return type: dict

>>> # noinspection PyUnresolvedReferences
>>> class SimpleConfig(Configurable):
...     def __init__(self, a=1, b='foo'):
...         self.a = a
...         self.b = b
...     def get_config(self):
...         return {'a': self.a, 'b': self.b}
>>> self = SimpleConfig()
>>> config = self.get_default_config()
>>> assert config == {'a': 1, 'b': 'foo'}

get_feedback_request(feedback_ids, feedback_type, test_id, round_id, session_id)[source]¶

Get Labels from the server based provided one or more example ids.

Parameters

feedback_ids (list) – List of media ids for which feedback is required
feedback_type (str) – Protocols constants with the values: label, detection, characterization
test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session

Return type

Returns

Path to a file containing containing requested feedback

classmethod get_impls()¶

Discover and return a set of classes that implement the calling class.

See the various discover_via_*() functions in this module for more details on the logic of how implementing classes (aka “plugins”) are discovered.

The class-level variables PLUGIN_ENV_VAR and PLUGIN_NAMESPACE may be overridden to change what environment and entry-point extension are looked for, respectively.

Return type: Set[Type[~P]]
Returns: Set of discovered class types that are considered “valid” plugins of this type. See is_valid_plugin() for what we define a “valid” type to be be relative to this class.

get_test_metadata(session_id, test_id)[source]¶

Retrieve the metadata json for the specified test.

Parameters

session_id (str) – The id of the session currently being evaluated
test_id (str) – The id of the test currently being evaluated

Return type

Returns

A dictionary containing metadata

classmethod is_usable()¶

Determine if this class with be detected by SMQTK’s plugin.

Return type: bool

post_results(result_files, test_id, round_id, session_id)[source]¶

Post client detector predictions for the dataset.

Parameters

result_files (Dict[str, str]) – A dictionary of results with protocol constant as key and file path as value
test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session

Return type

Returns

None

resume_session(session_id)[source]¶

Get finished test from an existing session.

Parameters

id (session) – Session id that was started but not terminated
session_id (str) –

Return type

List[str]

Returns

List of tests finished in the session

session_request(test_ids, protocol, domain, novelty_detector_version, hints, detection_threshold)[source]¶

Create a new session to evaluate the detector using an empirical protocol.

Parameters

test_ids (list) – List of tests being evaluated in this session
protocol (str) – String indicating which protocol is being evaluated
domain (str) – String indicating which domain is being evaluated
novelty_detector_version (str) – The novelty detector being evaluated
hints (list) – Hints used for the session
detection_threshold (float) – Detection threshold for the session

Return type

Returns

A session identifier provided by the server

terminate_session(session_id)[source]¶

Terminate the session after the evaluation for the protocol is complete.

Parameters: session_id (str) – The id provided by a server denoting a session
Return type: None

Returns: None

Return type: None
Parameters: session_id (str) –

test_ids_request(protocol, domain, detector_seed, test_assumptions='{}')[source]¶

Request Test Identifiers as part of a series of individual tests.

Parameters

protocol (str) – string indicating which protocol is being evaluated
domain (str) – problem domain for the tests
detector_seed (str) – A seed provided by the novelty detector
test_assumptions (str) – Assumptions used by the detector

Return type

Returns

Filename of file containing test ids

File Provider¶

Source code: sail_on_client/harness/file_provider.py

A provider class for evaluating with files.

class sail_on_client.harness.file_provider.FileProvider[source]¶

File-based service provider.

__init__(folder, results_folder)[source]¶

Initialize file provider.

Parameters

folder (str) – Folder where the data is present
results_folder (str) – Folder where the results are saved

Returns

None

Return type

None

complete_test(session_id, test_id)[source]¶

Mark test as completed in session logs.

Parameters

session_id (str) – The id of the session currently being evaluated
test_id (str) – The id of the test currently being evaluated

Return type

Returns

None

dataset_request(session_id, test_id, round_id)[source]¶

Request a dataset.

Parameters

test_id (str) – The test being evaluated at this moment.
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The identifier provided by the server for a single experiment

Return type

Optional[BytesIO]

Returns

Returns an instance of BytesIO

get_feedback(feedback_ids, feedback_type, session_id, test_id)[source]¶

Get feedback of the specified type.

Parameters

feedback_ids (List[str]) – List of media ids for which feedback is required
feedback_type (str) – Protocols constants with the values: label, detection, characterization
session_id (str) – The id provided by a server denoting a session
test_id (str) – The id of the test currently being evaluated

Return type

BytesIO

Returns

An instance of BytesIO with feedback

get_test_metadata(session_id, test_id, api_call=True, in_process_only=True)[source]¶

Get test metadata.

Parameters

session_id (str) – Session id for which the info is required
test_id (str) – Test id for which the info is required
api_call (bool) – Flag to change metadata to approved metadata
in_process_only (bool) – Flag to get information while the session is active

Return type

Returns

Metadata associated with the test as a dictionary

latest_session_info(session_id)[source]¶

Get tests finished from the most recent session.

Parameters: session_id (str) – The id of the session currently being evaluated
Return type: Dict
Returns: List of tests that have been marked as completed in the session

new_session(test_ids, protocol, domain, novelty_detector_version, hints, detection_threshold)[source]¶

Create a session.

Parameters

test_ids (List[str]) – Test ids evaluated in the session
protocol (str) – Name of the protocol
domain (str) – Domain of the protocol
novelty_detector_version (str) – Version of the novelty detector
hints (List[str]) – List of hints used in the session
detection_threshold (float) – Detection threshold used by the novelty detector

Return type

Returns

Session id for the agent

post_results(session_id, test_id, round_id, result_files)[source]¶

Post results.

Parameters

result_files (Dict[str, str]) – A dictionary of results with protocol constant as key and file path as value
test_id (str) – The id of the test currently being evaluated
round_id (int) – The sequential number of the round being evaluated
session_id (str) – The id provided by a server denoting a session

Return type

Returns

None

terminate_session(session_id)[source]¶

Terminate the session.

Parameters: session_id (str) – The id provided by a server denoting a session
Return type: None

Returns: None

Return type: None
Parameters: session_id (str) –

test_ids_request(protocol, domain, detector_seed, test_assumptions)[source]¶

Request test IDs.

Parameters

protocol (str) – Name of the protocol
domain (str) – Domain of the application
detector_seed (str) – Seed used by the detector
test_assumptions (str) – Assumptions made when generating the tests

Return type

Dict[str, str]

Returns

Dictionary with the list of tests and seed used for generating tests

File Provider Utility Methods¶

Source code: sail_on_client/harness/file_provider_fn.py

Set of functions used by file provider.

sail_on_client.harness.file_provider_fn.get_characterization_feedback(gt_file, result_files, feedback_ids, metadata)[source]¶

Calculate the proper feedback for characterization type feedback.

Parameters

gt_file (str) – Path to ground truth file
result_files (List[str]) – List of paths with results
feedback_ids (List[str]) – Element ids for which feedback is requested
metadata (Dict[str, Any]) – Dictionary with metadata

Return type

Returns

Dictionary containing feedback with feedback_ids as keys

sail_on_client.harness.file_provider_fn.get_classification_feedback(gt_file, result_files, feedback_ids, metadata)[source]¶

Calculate the proper feedback for classification type feedback.

Parameters

gt_file (str) – Path to ground truth file
result_files (List[str]) – List of paths with results
feedback_ids (List[str]) – Element ids for which feedback is requested
metadata (Dict[str, Any]) – Dictionary with metadata

Return type

Returns

Dictionary containing feedback with feedback_ids as keys

sail_on_client.harness.file_provider_fn.get_classificaton_score_feedback(gt_file, result_files, feedback_ids, metadata)[source]¶

Calculate feedback on the accuracy of classification.

Parameters

gt_file (str) – Path to ground truth file
result_files (List[str]) – List of paths with results
feedback_ids (List[str]) – Element ids for which feedback is requested
metadata (Dict[str, Any]) – Dictionary with metadata

Return type

Returns

Dictionary containing feedback with feedback_ids as keys

sail_on_client.harness.file_provider_fn.get_cluster_feedback(gt_file, result_files, feedback_ids, metadata)[source]¶

Calculate the proper feedback for cluster type feedback.

Parameters

gt_file (str) – Path to ground truth file
result_files (List[str]) – List of paths with results
feedback_ids (List[str]) – Element ids for which feedback is requested
metadata (Dict[str, Any]) – Dictionary with metadata

Return type

Returns

Dictionary containing feedback with feedback_ids as keys

sail_on_client.harness.file_provider_fn.get_encoding(domain)[source]¶

Get encoding based on the domain.

Parameters

domain (str) –
domain –

Return type

Returns

Encoding string for the domain

sail_on_client.harness.file_provider_fn.get_kinetics_labels_var_feedback(gt_file, result_files, feedback_ids, metadata)[source]¶

Get feedback for video activity recognition.

Parameters

gt_file (str) – Path to ground truth file
result_files (List[str]) – List of paths with results
feedback_ids (List[str]) – Element ids for which feedback is requested
metadata (Dict[str, Any]) – Dictionary with metadata

Return type

Returns

Dictionary containing feedback with feedback_ids as keys

sail_on_client.harness.file_provider_fn.get_levenshtein_feedback(gt_file, result_files, feedback_ids, metadata)[source]¶

Calculate the proper feedback for levenshtein type feedback.

Parameters

gt_file (str) – Path to ground truth file
result_files (List[str]) – List of paths with results
feedback_ids (List[str]) – Element ids for which feedback is requested
metadata (Dict[str, Any]) – Dictionary with metadata

Return type

Returns

Dictionary containing feedback with feedback_ids as keys

sail_on_client.harness.file_provider_fn.get_session_info(folder, session_id, in_process_only=True)[source]¶

Retrieve session info.

Parameters

folder (str) – Folder where the session file is saved
session_id (str) – Session id for which the info is required
in_process_only (bool) – Flag to get information while the session is active

Return type

Returns

Session information as a dict

sail_on_client.harness.file_provider_fn.get_session_test_info(folder, session_id, test_id)[source]¶

Retrieve session info for a test.

Parameters

folder (str) – Folder where the session file is saved
session_id (str) – Session id for which the info is required
test_id (str) – Test id for which the info is required

Return type

Returns

Session information associated with test as a dict

sail_on_client.harness.file_provider_fn.get_single_gt_feedback(gt_file, result_files, feedback_ids, metadata)[source]¶

Get feedback that specifies if a sample if correct or incorrect.

Parameters

gt_file (str) – Path to ground truth file
result_files (List[str]) – List of paths with results
feedback_ids (List[str]) – Element ids for which feedback is requested
metadata (Dict[str, Any]) – Dictionary with metadata

Return type

Returns

Dictionary containing feedback with feedback_ids as keys

sail_on_client.harness.file_provider_fn.log_session(folder, session_id, activity, test_id=None, round_id=None, content=None, content_loc='round', return_structure=False)[source]¶

Create a log files of all session activity.

Parameters

folder (str) – Folder where the session file is saved
session_id (str) – Session id associated with the log
test_id (Optional[str]) – Test id associated with the log
round_id (Optional[int]) – Round id associated with the log
content (Optional[Dict[str, Any]]) – Content of the log
content_loc (Optional[str]) – Location where the content should be added
return_structure (Optional[bool]) – Flag to return the updated log
activity (str) –

Return type

Optional[Dict]

Returns: Updated log if return_structure is set to True

Return type

Optional[Dict]

Parameters

folder (str) –
session_id (str) –
activity (str) –
test_id (Optional[str]) –
round_id (Optional[int]) –
content (Optional[Dict[str, Any]]) –
content_loc (Optional[str]) –
return_structure (Optional[bool]) –

sail_on_client.harness.file_provider_fn.psuedo_label_feedback(gt_file, feedback_ids, feedback_type, metadata, folder, session_id)[source]¶

Get psuedo label feedback for requested ids.

Parameters

gt_file (str) – Path to ground truth file
feedback_ids (List[str]) – Element ids for which feedback is requested
feedback_type (str) – Type of feedback
metadata (Dict[str, Any]) – Dictionary with metadata
folder (str) – Folder where the session file is saved
session_id (str) – Session id for which the info is required

Return type

Returns

Dictionary containing feedback with feedback_ids as keys

sail_on_client.harness.file_provider_fn.read_feedback_file(csv_reader, feedback_ids, metadata, check_constrained=True)[source]¶

Get feedback from feedback file for the specified ids in the last submitted round.

Parameters

csv_reader (reader) – An instance of csv reader
feedback_ids (Optional[List[str]]) – Element ids for which feedback is requested
metadata (Dict[str, Any]) – Dictionary with metadata
check_constrained (bool) – Flag to check constraints associated with feedback

Return type

Dict

Returns

Dictionary containing feedback with feedback_ids as keys

sail_on_client.harness.file_provider_fn.read_gt_csv_file(file_location)[source]¶

Read the ground truth csv file.

Parameters: file_location (str) – Path to the gt csv file
Return type: List
Returns: List of entries in csv file

sail_on_client.harness.file_provider_fn.read_meta_data(file_location)[source]¶

Read the metadata file.

Parameters: file_location (str) – Path to the gt csv file
Return type: Dict
Returns: Content of the metadata file

sail_on_client.harness.file_provider_fn.write_session_log_file(structure, filepath)[source]¶

Write session information in a log file.

Parameters

structure (Dict) – Dict with information about the session
filepath (str) – Path to the file where log file is saved

Return type