FEAT: Update evaluate_scorers

[varunj-msft]

Description

Creates a standardized scorer initialization pattern mirroring the existing AIRTTargetInitializer approach.

• Created pyrit/setup/initializers/components/ subdirectory
• Moved airt_targets.py → components/targets.py, renamed TargetConfig → AIRTTargetConfig
• Created components/scorers.py with AIRTScorerInitializer and 21 scorer configs
• Updated __init__.py exports for new module paths
• Refactored evaluate_scorers.py to use TargetRegistry for base targets and wire in both initializers

Tests and Documentation

• Added tests/unit/setup/test_airt_scorer_initializer.py
• Updated tests/unit/setup/test_airt_targets_initializer.py for new import paths

[Copilot]

Pull request overview

This PR introduces a standardized “component initializer” pattern for AIRT setup by adding a scorer initializer alongside the existing target initializer approach, and refactors the scorer-evaluation script to rely on registry-registered scorers.

Changes:

• Added pyrit/setup/initializers/components/ with dedicated target/scorer initializer modules and updated package exports.
• Introduced AIRTScorerInitializer with a centralized SCORER_CONFIGS list for evaluation scorers.
• Refactored build_scripts/evaluate_scorers.py to initialize via AIRT initializers and iterate scorers from ScorerRegistry.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/setup/test_airt_targets_initializer.py	Updates imports to the new components.targets module path.
tests/unit/setup/test_airt_scorer_initializer.py	Adds unit tests for AIRTScorerInitializer behavior and config coverage.
pyrit/setup/initializers/components/targets.py	New module defining TARGET_CONFIGS and AIRTTargetInitializer registration logic.
pyrit/setup/initializers/components/scorers.py	New module defining SCORER_CONFIGS and AIRTScorerInitializer registration logic.
pyrit/setup/initializers/components/init.py	Exposes component initializer types via __all__.
pyrit/setup/initializers/init.py	Re-exports AIRTScorerInitializer and updates AIRTTargetInitializer import path.
build_scripts/evaluate_scorers.py	Uses AIRTScorerInitializer + ScorerRegistry instead of hand-built scorer instances.

Comments suppressed due to low confidence (1)

build_scripts/evaluate_scorers.py:72

• Scorer.evaluate_async() defaults to update_registry_behavior=SKIP_IF_EXISTS, so this script may return cached metrics without re-running evaluations (and still prints “Evaluation complete and saved!”). If the intent is to benchmark scorers on each run, pass update_registry_behavior=RegistryUpdateBehavior.ALWAYS_UPDATE (and import the enum) or adjust the status messaging to reflect when cached results were used.

        try:
            print("  Status: Running evaluations...")
            results = await scorer.evaluate_async(
                num_scorer_trials=3,
                max_concurrency=10,
            )

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

build_scripts/evaluate_scorers.py:72

• This script now iterates over all registered scorers, but many registered scorers (e.g., TrueFalseCompositeScorer, FloatScaleThresholdScorer) don't configure evaluation_file_mapping, so evaluate_async() will raise ValueError for each of them. Consider filtering scorer_names up-front to only scorers with an evaluation mapping (or passing an explicit mapping), and optionally report which scorers were skipped due to missing evaluation datasets.

    registry = ScorerRegistry.get_registry_singleton()
    scorer_names = registry.get_names()

    if not scorer_names:
        print("No scorers registered. Check environment variable configuration.")
        return

    print(f"\nEvaluating {len(scorer_names)} scorer(s)...\n")

    # Use tqdm for progress tracking across all scorers
    scorer_iterator = (
        tqdm(enumerate(scorer_names, 1), total=len(scorer_names), desc="Scorers")
        if tqdm
        else enumerate(scorer_names, 1)
    )

    # Evaluate each scorer
    for i, scorer_name in scorer_iterator:
        scorer = registry.get_instance_by_name(scorer_name)
        print(f"\n[{i}/{len(scorer_names)}] Evaluating {scorer_name}...")
        print("  Status: Starting evaluation (this may take several minutes)...")

        start_time = time.time()

        try:
            print("  Status: Running evaluations...")
            results = await scorer.evaluate_async(
                num_scorer_trials=3,
                max_concurrency=10,
            )

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[rlundeen2]

looks good once addressing and resolving comments!