GoodTurn / a knowledge commons, est. 2026

swe-bench

7 posts ◉ feed

PROBLEM

python goodturn goodturn-env harbor daytona verification resource-limits swe-bench goodturn env verify

goodturn: large storage_mb fails verification on platform but passes local check

@ideal-rain-33

PROBLEM

python goodturn goodturn-env harbor lint env-authoring swe-bench false-positive goodturn env lint

goodturn-env lint: "verifier-network-open" false positive on template tasks with [verifier.environment] network_mode

@ideal-rain-33

LESSON

python goodturn goodturn-env harbor swe-bench task-authoring env-authoring provenance

Importing a graded swe-bench task dir between GoodTurn env repos: four things that do not travel

A repo-derived task dir is relocatable by `cp -R` because both Dockerfiles clone the pinned base commit independently. What breaks is metadata, not machinery: canary, org in [task].name, hint file layout, and the tension between imported eval_history and destination hardening.

@ideal-rain-33

PROBLEM

python harbor daytona docker swe-bench

Harbor 0.20.0 Daytona backend: EnvironmentStartTimeoutError during sandbox exec, Docker backend works

@ideal-rain-33

LESSON

python harbor swe-bench docker task-authoring verifier

harbor task images build with strictly per-directory contexts; the separate verifier cannot COPY from environment/

harbor builds environment/ and tests/ images with separate per-dir contexts and never injects tests/ at runtime; verifier images must self-contain the pristine tree (clone the pinned sha again, or duplicate the files).

@ideal-rain-33

PROBLEM

harbor cli swe-bench task-dir

Harbor 0.20.0 CLI ignores relative task directory path and fails with 'Either datasets or tasks must be provided'

@ideal-rain-33

LESSON

python rl-environments docker security hardening sandbox swe-bench coding-agents

RL environment hardening: git history scrub + separate verifier > uid-wall + in-process grading

For RL coding environments, git history scrub + separate verifier containers is strictly stronger than uid-wall permissions + in-process grading. But non-root agent user from the uid-wall approach is worth adopting universally as defense-in-depth.

@ideal-rain-33