Welcome to Sail-On Client’s documentation!

Release v0.33.0 (Installation)

Sail-On Client is a library used for supporting evaluation of open world algorithms in DARPA SAIL-ON. The library is based on the work supported by DARPA under Contract No. HR001120C0055 and is licensed under APACHE-V2 LICENSE.

Sail-On Client is being developed as an open source library to support evaluating vision centric open world agents. This involves communicating with evaluation server across a RESTful interface, providing empirical protocols that are used to systematically evaluate the agents capabilities across multiple levels of the novelty hierarchy.

Sail-On Client is built upon tinker-engine, an open source library that provides the abstractions and interface used for running the empirical protocols

Features

The major features provided by Sail-On Client

• RESTful interface for communicating with the evaluation server

• Empirical protocols for evaluating open world agents

• Dynamic algorithm discovery via plugins

• Configurable components with sensible defaults

• Saving and Restoring parameter for an algorithm

User Guide

• Installation
  • Requirements
  • Installation

Hydra

• Motivation For Using Hydra
• Configurable Components
• Command Line Interface
  • Usage
• Hydra Plugins
  • Introduction
  • Searchpath Plugin

Empirical Protocols

• Open World Novelty Detection (OND)
  • Introduction
  • Prediction Space
  • WorkFlow For OND
• Continuous Open-set Novelty Detection, Discovery, Adaptation (CONDDA)
  • Introduction
  • Prediction Space
  • WorkFlow For CONDDA
• Protocol API
  • Visual Protocol API
  • OND API
  • CONDDA API

Interfaces

• Harness
  • Local Harness
  • PAR Harness
• Harness API
  • Test And Evaluation Harness
  • Local Harness
  • PAR Harness
  • File Provider
  • File Provider Utility Methods

Metrics

• Program Metrics
  • Introduction
  • Symbols and Terms
  • M1: Average Number of FNs among Correctly Detected Trials (⬇ is better)
  • M2: Percentage of Correctly Detected Trials (⬆ is better)
  • M2.1: Percentage of False Positive Trials (⬇ is better)
  • M3 and M4: Novelty Reaction Performance (⬆ is better)
  • M5: Overall Performance Task Improvement (⬆ is better)
  • M6: Asymptotic Performance Task Improvement (⬆ is better)
• Metric API
  • Methods For Metric
  • Base Class For Metric
  • Activity Recognition Metric
  • Document Transcription Metric
  • Image Classification Metric
  • Utility Methods

Feedback

• Feedback
  • Introduction
  • Feedback Across Domains
• Feedback API
  • Base Class for Feedback
  • Activity Recognition Feedback
  • Document Transcription Feedback
  • Image Classification Feedback

Agents

• Agents
  • Introduction
  • Functions Used By the Protocol
  • Adapters
  • Sample Detector and Registration
• Agents API
  • Visual Agent
  • OND Agents
  • Mock OND Agents
  • Pre Computed Agents
  • CONDDA Agents
  • Mock CONDDA Agents

Saving and Restoring Models

• Saving And Restoring Features for Algorithm
  • Introduction
  • Config Options For Saving Features
  • Config Options For Using Features
• Saving And Restoring Attributes of Algorithm
  • Introduction
  • Sample Detector and Adapter
• Checkpoint API
  • Checkpointer Mixin

Indices and tables

• Index

• Module Index

• Search Page