New bucket for team Thoth [Superset, Hue]

[xtuchyna]

Questionnaire

1. User details:
   MOC username: [email protected]
   Applications to be used with bucket: most recent versions of Hue and Superset (I think that Superset is at 1.0.0 now?) that are possible to deploy and are compatible with themselves.
   Bucket should be public (as e.g. DH-PLAYPEN on ODH Hue)

2. Desired bucket name:
   thoth-public

3. Maximal bucket size:
   2GiB

4. Maximal bucket object count:
   10 000

5. Your GPG key or contact:
   My GPG key:
   dominiktuchyna_gpg.zip

If you don't want to wait for the maintainers to notice this issue, you can do the changes yourself. Please follow the docs/user_profiles.md guide.

[tumido]

Hi @xtuchyna sorry for a late response. We have a planned cluster tear down on Monday followed by a new cluster provisioning therefore we're not that quick to onboard new requests.

If you want to use the bucket in your own project (meaning you have a namespace available to you), you can just deploy a ObjectBucketClaim resource as described here.

If you don't have a namespace of your own, we can do that for you on Monday.

Please be advised that any ObjectBucketClaims are hosted on the cluster and its availability is limited to the cluster lifespan. If you want to get a more permanent and resilient storage (one that is not deleted with our cluster) I suggest you choose to follow this doc: open-infrastructure-labs/ops-issues#33

[xtuchyna]

Hello @tumido , I don't have namespace on my own in CNV cluster, so if I may ask you to create one for me?

[xtuchyna]

update: I will check with team what namespace do we need and than let you know

[harshad16]

We are just looking for a public bucket, is operate-first having such a bucket? or should we proceed by creating an objectbucketclaim?

[harshad16]

This is in regards to using the superset and hue in operate-first

[xtuchyna]

Hello @tumido , please what's the current status of this?

[tumido]

I'm really sorry @xtuchyna it took me so long, I had a lot on my plate lately and I kinda stalled this ticked. Thank you for the ping! 🙂

I'm worried we will have problems with creating public buckets. You may need to apply a policy later on which would make the bucket public (or you can share the credentials with the world). By default we can't create buckets which are available with anonymous access... I'll be creating the bucket for you today and will keep you posted. 👍

[xtuchyna]

Okay!! thank you very much and I am looking forward to your reply 💯

[tumido]

Hey @xtuchyna a new bucket should be created for you:

Bucket name: opf-datacatalog-dtuchyna
Endpoint: s3.openshift-storage.svc:443 internally within the cluster, https://s3-openshift-storage.apps.zero.massopen.cloud/
Credentials: xtuchyna.txt (it's asc, but GH doesn't like that extension)

let me know if it works 👍

[xtuchyna]

Hello @tumido , I've tried to access it but I am probably doing something wrong?

[xtuchyna@c01-424a ~]$ aws s3 ls --profile opf s3://opf-datacatalog-dtuchyna/

An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS access key Id you provided does not exist in our records.

Here's my config:

[xtuchyna@c01-424a ~]$ cat .aws/config 
[default]
profile = upshift

[profile upshift]
#region = us-east-1
#output = json
s3 =
    endpoint_url = https://s3.upshift.redhat.com/

[profile opf]
#region = us-east-1
#output = json
s3 =
    endpoint_url = https://s3-openshift-storage.apps.zero.massopen.cloud/

[plugins]
endpoint = awscli_plugin_endpoint

(the opf profile credentials are the ones you've sent me)

[xtuchyna]

Hello @tumido, so both me and @pacospace tried the credentials, but most probably they are invalid (see comment above)

[pacospace]

Hello @tumido, so both me and @pacospace tried the credentials, but most probably they are invalid (see comment above)

the AWS_KEY_ACCESS_KEY_ID are in capital letters usually, in the credentials @xtuchyna has received, they are not.

[tumido]

the AWS_KEY_ACCESS_KEY_ID are in capital letters usually

@pacospace that is true for AWS only. Ceph is issuing credentials differently. The creds I gave you should match the ones in the secret, let's compare what you have:

@xtuchyna I was able to use it with s3cmd just fine:

s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key []: tim...
Secret Key []: TdRd...                                      
Default Region [US]: 

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint []: s3-openshift-storage.apps.zero.massopen.cloud

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket []: s3-openshift-storage.apps.zero.massopen.cloud/%(bucket)s

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: 

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: 

New settings:
  Access Key: tim...
  Secret Key: TdR...
  Default Region: US
  S3 Endpoint: s3-openshift-storage.apps.zero.massopen.cloud
  DNS-style bucket+hostname:port template for accessing a bucket: s3-openshift-storage.apps.zero.massopen.cloud/%(bucket)s
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/home/tcoufal/.s3cfg'

$ s3cmd ls s3://opf-datacatalog-dtuchyna

$ touch test.txt

$ s3cmd put test.txt s3://opf-datacatalog-dtuchyna/test.txt
upload: 'test.txt' -> 's3://opf-datacatalog-dtuchyna/test.txt'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

$ s3cmd ls s3://opf-datacatalog-dtuchyna/                 
2021-03-23 11:26            0  s3://opf-datacatalog-dtuchyna/test.txt

These are the important bits from my .s3cfg:

[default]
access_key = tim...
host_base = s3-openshift-storage.apps.zero.massopen.cloud
host_bucket = s3-openshift-storage.apps.zero.massopen.cloud/%(bucket)s
secret_key = TdR...
use_https = True

I don't have aws cli on me, but it should work as well. I suspect you have to use endpoint_url=s3-openshift-storage.apps.zero.massopen.cloud without that explicit https://

[xtuchyna]

therefore I think the creds does not match?

[xtuchyna@c01-424a Downloads]$ gpg --decrypt xtuchyna.asc 
gpg: encrypted with 4096-bit RSA key, ID D2471090FB182D93, created 2020-05-12
      "Dominik Tuchyna <[email protected]>"
AWS_ACCESS_KEY_ID: dGl...
AWS_SECRET_ACCESS_KEY: VGR...

[tumido]

Ok.. 😄 Then what have I sent you? I have no idea where those credentials I've sent you came from then... They don't exist anywhere on the cluster. We may monitor this, because it feels like the credentials were rotated(?) If that's true, that's not good.

Anyways, here's the proper set of credentials for you and I humbly apologize for this mess. I'm very sorry, sir!

dtuchyna.gpg.txt

[xtuchyna]

nice, it works :) ! thanks a lot @tumido 🍻

[pacospace]

the AWS_KEY_ACCESS_KEY_ID are in capital letters usually

@pacospace that is true for AWS only. Ceph is issuing credentials differently. The creds I gave you should match the ones in the secret, let's compare what you have:

@xtuchyna I was able to use it with s3cmd just fine:

s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key []: tim...
Secret Key []: TdRd...                                      
Default Region [US]: 

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint []: s3-openshift-storage.apps.zero.massopen.cloud

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket []: s3-openshift-storage.apps.zero.massopen.cloud/%(bucket)s

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: 

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: 

New settings:
  Access Key: tim...
  Secret Key: TdR...
  Default Region: US
  S3 Endpoint: s3-openshift-storage.apps.zero.massopen.cloud
  DNS-style bucket+hostname:port template for accessing a bucket: s3-openshift-storage.apps.zero.massopen.cloud/%(bucket)s
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/home/tcoufal/.s3cfg'

$ s3cmd ls s3://opf-datacatalog-dtuchyna

$ touch test.txt

$ s3cmd put test.txt s3://opf-datacatalog-dtuchyna/test.txt
upload: 'test.txt' -> 's3://opf-datacatalog-dtuchyna/test.txt'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

$ s3cmd ls s3://opf-datacatalog-dtuchyna/                 
2021-03-23 11:26            0  s3://opf-datacatalog-dtuchyna/test.txt

These are the important bits from my .s3cfg:

[default]
access_key = tim...
host_base = s3-openshift-storage.apps.zero.massopen.cloud
host_bucket = s3-openshift-storage.apps.zero.massopen.cloud/%(bucket)s
secret_key = TdR...
use_https = True

I don't have aws cli on me, but it should work as well. I suspect you have to use endpoint_url=s3-openshift-storage.apps.zero.massopen.cloud without that explicit https://

Thanks a lot @tumido!! Good to know :) 🍻