[WIP] s3Opener: add query params for mixed and non-aws s3 resources

[RussTorres]

This adds support for s3 uris using custom profile/region/endpoints via query parameters. Our use case likely involves multiple non-aws s3 sources, which the current s3opener does not support.

I marked this as WIP as I have not really seen people use query parameters in s3 uris this way, but it seems that parsing these options in a uri would be the logical way to point somewhere. I am open to other ideas and would be happy to implement the other config/credential arguments into the uri somewhere in another commit.

[RussTorres]

The other way I have seen folks do this is with a custom protocol, so we could have a configuration in our server deployment that makes a url handler for something like mys3system://mybucket/path/to/my/key. This is in line with what cloudvolume has hardcoded seung-lab/cloud-volume#170

[trautmane]

Hi @RussTorres . At first glance, I think I prefer the custom protocol names with configured credential (profile, endpoint, ...) mapping approach. You could setup an s3 handler factory similar to the image filter factory ( https://github.com/saalfeldlab/render/blob/master/render-app/src/main/java/org/janelia/alignment/filter/FilterFactory.java ) and then reference that in the S3Opener. One place this might complicate things though is with client-side rendering since the configured mappings would need to also be available on the client. Is that relevant for your "custom" s3 use cases?

@perlman - any thoughts?

[RussTorres]

Yes, the clients will need access to the images, but I have deployments with custom filtering working, so I am behind the urihandler factory -- it also adds a layer where we are not dependent on write-specified metadata in cases of, e.g. DNS failure.

[perlman]

This is reminding me how ugly the ImageJ Opener is.

These modifications seem ok to me, as an extension of the current kludge, but I agree that a more abstract approach would be nice. For example, I could see implementing an opener that looks in memcache or redis first.

My first pass for handling S3 was through registering a handler in java.protocol.handler.pkg. I recall it working, but being quite brittle.