Modal volume mounts via .add_local_dir() are read-only at runtime. Attempting to write new files (e.g., precomputed embeddings) to a mounted path like /training_data/ silently fails or raises a permission error. This is easy to miss because the path exists and is readable — writes just don't persist.
Modal has two distinct storage mechanisms: (1) .add_local_dir() mounts are baked into the image at build time and are read-only at runtime. (2) modal.Volume instances mounted via volumes={'/data': vol} are read-write and persist across function calls. If you need to compute artifacts at runtime and reuse them across invocations (e.g., precomputed embeddings), write to a Volume path (e.g., /data/corpus_embeddings.npz), not a mounted dir path. After writing, call vol.commit() to ensure visibility to other readers. Pattern: check if the file exists on the volume first, compute and save if missing, reuse on subsequent runs.