-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
Describe the feature you'd like
In the SDK v2, ProcessingOutput has a destination parameter to be either an S3 path where users want to write their results, but they can set it to None when they want to delegate the storage to SageMaker. This option stores the results in an S3 prefix partitioned by job_name, step_name, and output_name, which is quite convenient for traceability and to store intermediate results of a pipeline.
How would this feature be used? Please describe.
Much like v2, v3 should also support this feature. Users may not want to fix the destination path for intermediate results, so ideally the same behaviour should apply to v3 by leaving ProcessingS3Output's s3_uri field as None. Currently this field is mandatory and it's unclear whether users can reproduce the v2 None behaviour.
Describe alternatives you've considered
No alternatives considered so far other than sticking to v2.
Additional context
N/A