Page 1 of 1

OpenNyAI: Lawyer Registration for Ground Truth Data

About

OpenNyAI is testing out data retrieval systems for an open-source tool designed for lawyers and judges looking for quick access to sections of bare acts, amendments, etc. and to run subjective queries requiring analysis and synthesis across either multiple sections of an Act or multiple sections of multiple acts.
To test a retrieval system on these tasks, we need an objective set of data that is sourced from human experts (lawyers and judges) that we can test the system against.

How you can participate

Some example data sets are provided below. The dataset comprises of sample questions and expected correct answers that an LLM should provide. Below are also examples of broad categories of questions to consider. Subject to your participation in this exercise, you will be asked to submit a set of minimum 10 question and answer pairs that cut across the provided categories.

Sample Questions & Answer Pairs

https://storage.tally.so/b8ef1a3c-3a87-4f33-9e78-1e7340b4fd0e/Screenshot-2023-10-25-at-2.34.18-PM.png

Sample Classifications of Queries

Objective Queries:

1. Section Retrieval from Bare Act

2. Definition Clauses, as provided for in an Act

3. Jurisdiction of an Act 

4. Amendments to an Act

5. Related Rules, Notifications and Regulations to an Act

Subjective:

1. Comparison of Definitions across Acts

2. Open-ended questions regarding the powers and punishments.

Registration