English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In his "ship of state" allegory (\textit{Republic}, Book VI, 488) Plato posesa question -- how can a crew of sailors presumed to know little about the artof navigation recognize the true pilot among them? The allegory argues that asimple majority voting procedure cannot safely determine who is most qualifiedto pilot a ship when the voting members are ignorant or biased. We formalizePlato's concerns by considering the problem in AI safety of monitoring noisy AIagents in unsupervised settings. An algorithm evaluating AI agents usingunlabeled data would be subject to the evaluation dilemma - how would we knowthe evaluation algorithm was correct itself? This endless validation chain canbe avoided by considering purely algebraic functions of the observed responses.We can construct complete postulates than can prove or disprove the logicalconsistency of any grading algorithm. A complete set of postulates existswhenever we are evaluating $N$ experts that took $T$ tests with $Q$ questionswith $R$ responses each. We discuss evaluating binary classifiers that havetaken a single test - the $(N,T=1,Q,R=2)$ tests. We show how some of thepostulates have been previously identified in the ML literature but notrecognized as such - the \textbf{agreement equations} of Platanios. Thecomplete postulates for pair correlated binary classifiers are considered andwe show how it allows for error correlations to be quickly calculated. Analgebraic evaluator based on the assumption that the ensemble is errorindependent is compared with grading by majority voting on evaluations usingthe \uciadult and and \texttt{two-norm} datasets. Throughout, we demonstratehow the formalism of logical consistency via algebraic postulates of evaluationcan help increase the safety of machines using AI algorithms.

The logic of NTQR evaluations of noisy AI agents: Complete postulates and logically consistent error correlations