Feedback API

Base Class for Feedback

Source code: sail_on_client/feedback/feedback.py

Abstract class for feedback for sail-on.

class sail_on_client.feedback.feedback.Feedback[source]

Base class for Feedback.

__init__(first_budget, income_per_batch, maximum_budget, interface, session_id, test_id, feedback_type)[source]

Initialize.

Parameters

  • first_budget (int) – Initial budget

  • income_per_batch (int) – Additional labels added after every batch

  • maximum_budget (int) – Max labels that can be requested

  • interface (Union[LocalHarness, ParHarness]) – An instance of evaluation interface

  • session_id (str) – Session identifier

  • test_id (str) – Test identifier

  • feedback_type (str) – Type of feedback that can be requested

Returns

None

Return type

None

deposit_income()[source]

Get income for a round.

Return type

None

get_budget()[source]

Get current budget.

Return type

int

get_feedback(round_id, images_id_list, image_names)[source]

Get feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Union[DataFrame, Dict, None]

Returns

Either a dataframe or dictionary with score if the request is valid for the current round.

get_labeled_feedback(round_id, images_id_list, image_names)[source]

Get labeled feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[DataFrame]

Returns

A dictionary with the accuracy value or None if feedback is requested for an older round

get_score_feedback(round_id, images_id_list, image_names)[source]

Get accuracy value for the round. Note: this is not budgeted.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[Dict]

Returns

A dictionary with the accuracy value or None if feedback is requested for an older round

Activity Recognition Feedback

Source code: sail_on_client/feedback/activity_recognition_feedback.py

Activity Recognition Feedback.

class sail_on_client.feedback.activity_recognition_feedback.ActivityRecognitionFeedback[source]

Feedback for activity recognition.

__init__(first_budget, income_per_batch, maximum_budget, interface, session_id, test_id, feedback_type)[source]

Initialize activity recognition feedback object.

Parameters

  • first_budget (int) – Initial budget

  • income_per_batch (int) – Additional labels added after every batch

  • maximum_budget (int) – Max labels that can be requested

  • interface (Union[LocalHarness, ParHarness]) – An instance of evaluation interface

  • session_id (str) – Session identifier

  • test_id (str) – Test identifier

  • feedback_type (str) – Type of feedback that can be requested

Returns

None

Return type

None

deposit_income()

Get income for a round.

Return type

None

get_budget()

Get current budget.

Return type

int

get_classification_feedback(round_id, images_id_list, image_names)[source]

Get labeled feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[DataFrame]

Returns

A dictionary with the accuracy value or None if feedback is requested for an older round

get_detection_and_classification_feedback(round_id, images_id_list, image_names)[source]

Get detection and classification feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List of indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[DataFrame]

Returns

A dataframe with id and novelty detection value as columns

get_detection_feedback(round_id, images_id_list, image_names)[source]

Get detection feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List of indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[DataFrame]

Returns

A dataframe with id and novelty detection value as columns

get_feedback(round_id, images_id_list, image_names)[source]

Get feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Union[DataFrame, Dict, None]

Returns

Either a dataframe or dictionary with score if the request is valid for the current round.

get_labeled_feedback(round_id, images_id_list, image_names)

Get labeled feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[DataFrame]

Returns

A dictionary with the accuracy value or None if feedback is requested for an older round

get_score_feedback(round_id, images_id_list, image_names)

Get accuracy value for the round. Note: this is not budgeted.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[Dict]

Returns

A dictionary with the accuracy value or None if feedback is requested for an older round

Document Transcription Feedback

Source code: sail_on_client/feedback/document_transcription_feedback.py

Document Transcription Feedback.

class sail_on_client.feedback.document_transcription_feedback.DocumentTranscriptionFeedback[source]

Feedback for document transcription.

__init__(first_budget, income_per_batch, maximum_budget, interface, session_id, test_id, feedback_type)[source]

Initialize document transcription feedback object.

Parameters

  • first_budget (int) – Initial budget

  • income_per_batch (int) – Additional labels added after every batch

  • maximum_budget (int) – Max labels that can be requested

  • interface (Union[LocalHarness, ParHarness]) – An instance of evaluation interface

  • session_id (str) – Session identifier

  • test_id (str) – Test identifier

  • feedback_type (str) – Type of feedback that can be requested

Returns

None

Return type

None

deposit_income()

Get income for a round.

Return type

None

get_budget()

Get current budget.

Return type

int

get_feedback(round_id, images_id_list, image_names)[source]

Get feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Union[DataFrame, Dict, None]

Returns

Either a dataframe or dictionary with score if the request is valid for the current round.

get_labeled_feedback(round_id, images_id_list, image_names)

Get labeled feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[DataFrame]

Returns

A dictionary with the accuracy value or None if feedback is requested for an older round

get_levenshtein_feedback(round_id, images_id_list, image_names)[source]

Get levenshtein feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[Dict]

Returns

A dictionary containing levenshtein score or None if feedback is requested for an older round

get_score_feedback(round_id, images_id_list, image_names)

Get accuracy value for the round. Note: this is not budgeted.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[Dict]

Returns

A dictionary with the accuracy value or None if feedback is requested for an older round

Image Classification Feedback

Source code: sail_on_client/feedback/image_classification_feedback.py

Image Classification Feedback.

class sail_on_client.feedback.image_classification_feedback.ImageClassificationFeedback[source]

Feedback for image classification.

__init__(first_budget, income_per_batch, maximum_budget, interface, session_id, test_id, feedback_type)[source]

Initialize image classification feedback object.

Parameters

  • first_budget (int) – Initial budget

  • income_per_batch (int) – Additional labels added after every batch

  • maximum_budget (int) – Max labels that can be requested

  • interface (Union[LocalHarness, ParHarness]) – An instance of evaluation interface

  • session_id (str) – Session identifier

  • test_id (str) – Test identifier

  • feedback_type (str) – Type of feedback that can be requested

Returns

None

Return type

None

deposit_income()

Get income for a round.

Return type

None

get_budget()

Get current budget.

Return type

int

get_feedback(round_id, images_id_list, image_names)

Get feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Union[DataFrame, Dict, None]

Returns

Either a dataframe or dictionary with score if the request is valid for the current round.

get_labeled_feedback(round_id, images_id_list, image_names)

Get labeled feedback for the round.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[DataFrame]

Returns

A dictionary with the accuracy value or None if feedback is requested for an older round

get_score_feedback(round_id, images_id_list, image_names)

Get accuracy value for the round. Note: this is not budgeted.

Parameters

  • round_id (int) – Round identifier

  • image_id_list – List if indices for images

  • image_names (list) – List of image names for the round

  • images_id_list (list) –

Return type

Optional[Dict]

Returns

A dictionary with the accuracy value or None if feedback is requested for an older round