Install s3cmd

The S3 API allows developers to interact with S3 storage resources programmatically, enabling them to create, manage, and manipulate objects (files) in S3 buckets (containers) using various operations.

The S3 API is RESTful (Representational State Transfer) in nature, which means it follows the principles of the REST architecture and uses standard HTTP methods, such as GET, PUT, POST, DELETE, etc., to perform operations on S3 objects and buckets. The S3 API supports both synchronous and asynchronous operations, allowing developers to interact with S3 in real-time or perform batch operations as needed.

The s3 API requires install and configuration. Follow the steps below to get started:

Install

To install run the apt install as shown below.

sudo apt install s3cmd

The system will provide warning of packages that'll be installed and size requirements. When prompted type 'y -> enter' to continue.

The following NEW packages will be installed:
  python3-dateutil s3cmd
0 upgraded, 2 newly installed, 0 to remove and 1 not upgraded.
Need to get 199 kB of archives.
After this operation, 861 kB of additional disk space will be used.
Do you want to continue? [Y/n]

If running homebrew for Mac use brew install.

brew install s3cmd

Configure s3cmd

URL and API Keys

The URL and API keys are required for configuration. Follow the steps below to configure s3cmd with the American Cloud CMP.

Sign into American Cloud CMP
On the left navigation column choose 'Storage'
Choose the A2 Object Storage header tab select 'Manage'

Endpoint URL

The S3 API endpoint URL is the web address used to interact with Amazon S3 programmatically. It specifies the S3 service region and provides a RESTful interface for performing operations on S3 objects and buckets via HTTP/HTTPS requests. The endpoint should follow the below configuration.

Select Object Storage 'Settings'
Select 'Keys' inside settings menu

API Keys

S3 API keys are access credentials that enable programmatic interaction with Amazon S3. They consist of an access key and a secret access key, and are used to authenticate requests to perform operations on S3 objects and buckets via the S3 API.

Configure Using s3 Wizard

The S3 configuration wizard is a tool that guides users through the process of setting up and configuring an Object Storage bucket. It provides step-by-step instructions for configuring access permissions, encryption, and other settings for the bucket.- Start the configurer by entering the below command. Below are the steps to configure with American Cloud's A2 Object Storage.

Start the Wizard

Once install, s3 commands will be recognized. To start the configuration wizard run the following:

s3cmd --configure

Once started a readout like below will populate asking for the access key.

cloud@Compute-1:~$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key:

Key Placement

Identification of American Object Storage Keys was explained in previous steps. Easily copy/paste the keys in the respective location as requested by the configuration wizard. [US] for connecting can be left as default.

Access Key: EXAMPLEJYJGRYBV6X
Secret Key: EXAMPLEIuxqWjBad31hjQi3Eo97YM4
Default Region [US]:

Endpoint URL

Identification of American Object Storage endpoint URL was explained in previous steps. Easily copy/paste the American Cloud endpoint. Below example is for buckets within American Cloud West region. This may change for some.

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: a2-west.americancloud.com

DNS

The S3 configuration wizard prompts for a URL template to access the bucket. Using the variable %(bucket)s. For this example in American Cloud object storage place 'n' for NO and press enter.

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:

Encryption Password

The S3 configuration wizard may prompt for an optional encryption password. GPG encryption protects files both in transit and while stored on American Cloud's A2 Object Storage, unlike HTTPS which only protects files in transit.

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:

Path to GPG

If using GPG a path to is required in the next prompt. On linux machines leave the default.

Path to GPG program:

HTTPS

Next the wizard will prompt to use HTTPS which will protect traffic while being transmitted. Press enter to except the default of [Yes].

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

Proxy

Leave blank and press enter unless running proxy. If so place IP or domain here.

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

Validate and Test

The wizard will provide an overview of all newly assigned settings. Validate all settings are correct and input 'Y' to run test with provided settings.

New settings:
Access Key:
Secret Key:
Default Region: US
S3 Endpoint: s3.amazonaws.com
DNS-style bucket+hostname: n
Encryption password:
Path to GPG program: None
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]

Save

Next prompt will ask if saving is desired. If yes type 'Y' press enter. If settings are saved the settings will be placed within the .s3cfg file for further use. If no the settings will be deleted.

Save settings? [y/N]

Retry

The wizard next provides the option to retry the configuration. Select between Y/n.

Retry configuration? [Y/n]

As previously mentioned, if chosen to save the configuration settings in the wizard, the settings will be stored in the .s3cfg file. If changes to the configuration settings are neccessary, such as generating new keys, easily access the .s3cfg file by running the command vi .s3cfg in your terminal or command prompt, and then edit the document accordingly.

[default]
access_key = ACCESS KEY HERE
secret_key = SECRET KEY HERE
access_token =
add_encoding_exts =
add_headers =
bucket_location = US
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
connection_max_age = 5
connection_pooling = True
content_disposition =
content_type =
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = None
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = a2-west.americancloud.com
host_bucket = a2-west.americancloud.com
human_readable_sizes = True
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limit = -1
limitrate = 0
list_allow_unordered = False
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunk_size_mb = 500
multipart_copy_chunk_size_mb = 2048
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
public_url_use_https = True
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
restore_priority = Standard
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
signurl_use_https = True
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
ssl_client_cert_file =
ssl_client_key_file =
stats = False
stop_on_error = False
storage_class =
throttle_max = 100
upload_id =
urlencoding_mode = normal
use_http_expect = False
use_https = True
use_mime_magic = True
verbosity = WARNING
website_endpoint = https://a2-west.americancloud.com/
website_error =
website_index = index.html`^`

Add Buckets

Object storage buckets are containers for storing and organizing large volumes of unstructured data, such as files, images, and videos, in the cloud. They provide scalable, durable, and cost-effective storage solutions, allowing users to upload, retrieve, and manage data using APIs or web interfaces. Below are list of commands for adding buckets.

Make Bucket Command

# Use mb (make bucket) command
s3cmd mb s3://americancloud-1

cloud@Compute-1:~$ s3cmd mb s3://americancloud-1
Bucket 's3://americancloud-1/' created

Using the ls command list buckets.

s3cmd ls

The new bucket is listed.

cloud@Compute-1:~$ s3cmd ls
2023-04-19 23:46  s3://americancloud-1
2023-04-19 21:02  s3://bucketac2

As expected the bucket has been placed inside AC CMP.

Removing a Bucket

Removing buckets is a process that permanently deletes a bucket and all its objects. To remove a bucket, the user must have appropriate permissions, and all objects within the bucket must be deleted first. Once a bucket is removed, its data cannot be recovered. It is important to exercise caution and ensure backups are in place before deleting any buckets in S3.

Remove Bucket Command

s3cmd rb

cloud@Compute-1:~$ s3cmd rb s3://ac-123
Bucket 's3://ac-123/' removed

CMP side the bucket has been removed as well.

List Buckets and Files

Listing files in S3 involves retrieving a list of objects (files) stored within a specific bucket. The list typically includes information such as object names, sizes, and metadata. It can be useful for navigating and managing objects in S3, including copying, deleting, or downloading files. Proper access permissions and authentication are required to list files in S3, ensuring data security and privacy.

List command

The ls command will list the buckets within Object Storage.

s3cmd ls

cloud@Compute-1:~$ s3cmd ls
2023-04-19 23:46  s3://americancloud-1
2023-04-20 17:51  s3://americancloud-2

List files within a bucket by running the ls s3://*bucketname.

s3cmd ls s3://americancloud-1

cloud@Compute-1:~$ s3cmd ls s3://americancloud-1
2023-04-20 03:16 89.6904296875k  s3://americancloud-1/AC is Awesome.pages
2023-04-20 00:39 148.197265625k  s3://americancloud-1/Screenshot 2023-04-19 at 6.52.20 PM-20230420123958.png
2023-04-20 00:20 148.90234375k  s3://americancloud-1/Screenshot 2023-04-19 at 7.19.54 PM-20230420122021.png
2023-04-20 17:15         0   s3://americancloud-1/americancloudisawesome.txt
2023-04-20 17:14         0   s3://americancloud-1/sample.txt

Additionally, list all files within all buckets by executing s3cmd la.

s3cmd la

cloud@Compute-1:~$ s3cmd la
2023-04-20 03:16 89.6904296875k  s3://americancloud-1/AC is Awesome.pages
2023-04-20 00:39 148.197265625k  s3://americancloud-1/Screenshot 2023-04-19 at 6.52.20 PM-20230420123958.png
2023-04-20 00:20 148.90234375k  s3://americancloud-1/Screenshot 2023-04-19 at 7.19.54 PM-20230420122021.png
2023-04-20 17:15         0   s3://americancloud-1/americancloudisawesome.txt
2023-04-20 17:14         0   s3://americancloud-1/sample.txt

2023-04-20 17:52         0   s3://americancloud-2/americancloudisawesome.txt

Add Files

The "put" command in S3 is a command-line operation that allows users to upload (put) objects (files) from their local system to an S3 bucket. The "put" command requires specifying the source file path, destination S3 bucket name, and object key (file name) to store the object in S3. Proper permissions and authentication are necessary for successful object uploads.

PUT Command

Single file upload

s3cmd put /file s3://americancloud-1

s3cmd put americancloudisawesome.txt s3://americancloud-1
upload: 'americancloudisawesome.txt' -> 's3://americancloud-1/americancloudisawesome.txt'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

Multiple file upload

s3cmd put ac1.txt ac2.txt path/to/ac3.txt s3://americancloud-1

s3cmd put acisawesome.txt americancloudisawesome.txt s3://bucketac4
upload: 'acisawesome.txt' -> 's3://bucketac4/acisawesome.txt'  [1 of 2]
 0 of 0     0% in    0s     0.00 B/s  done
upload: 'americancloudisawesome.txt' -> 's3://bucketac4/americancloudisawesome.txt'  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done

Change name during upload

s3cmd put ac1.txt s3://americancloud-1/newname.txt

s3cmd put test.txt s3://bucketac4/ac-2.txt
upload: 'test.txt' -> 's3://bucketac4/ac-2.txt'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

If desired an entire director can be moved using 'sync' command. Idea for backup scenarios

cloud@Compute-AC-9:~$ s3cmd sync /home/cloud s3://bucketac4
upload: '/home/cloud/.bash_history' -> 's3://bucketac4/cloud/.bash_history'  [1 of 12]
 0 of 0     0% in    0s     0.00 B/s  done
upload: '/home/cloud/.bash_logout' -> 's3://bucketac4/cloud/.bash_logout'  [2 of 12]
 220 of 220   100% in    0s     7.99 KB/s  done
upload: '/home/cloud/.bashrc' -> 's3://bucketac4/cloud/.bashrc'  [3 of 12]

Retrieving Files

To retrieve files in S3, a cloud-based object storage service, you can use the S3 API or S3 console. First, authenticate and authorize access, then specify the S3 bucket and object key to identify the file. Use the appropriate method, such as GET, to retrieve the file from S3. Optionally, you can configure access control and encryption settings for added security.

GET Command

Single file download

s3cmd get s3://[bucketname]/filename

s3cmd get s3://bucketac4/ac-2.txt
download: 's3://bucketac4/ac-2.txt' -> './ac-2.txt'  [1 of 1]
0 of 0     0% in    0s     0.00 B/s  done

Multiple file download

s3cmd get s3://bucketac4/test1.txt s3://bucketac4/test2.txt
download: 's3://bucketac4/test1.txt' -> './test1.txt'  [1 of 2]
0 of 0     0% in    0s     0.00 B/s  done
download: 's3://bucketac4/test2.txt' -> './test2.txt'  [2 of 2]
0 of 0     0% in    0s     0.00 B/s  done

Change file name

s3cmd get s3://[bucketname]/filename newfilename

s3cmd get s3://bucketac4/ac-4.txt ac-5.txt --recursive
download: 's3://bucketac4/ac-4.txt' -> 'ac-5.txt'  [1 of 1]
0 of 0     0% in    0s     0.00 B/s  done

Use of --recursive. To pull all files from a bucket use the recursive flag.

s3cmd get s3://[bucketname]/ --recursive

s3cmd get s3://bucketac4/ --recursive
download: 's3://bucketac4/Screenshot 2023-04-18 at 11.03.46 PM-20230423120714.png' -> './Screenshot 2023-04-18 at 11.03.46 PM-20230423120714.png'  [1 of 10]
512226 of 512226   100% in    0s  1229.82 KB/s  done
download: 's3://bucketac4/Screenshot 2023-04-21 at 5.20.19 PM-20230423120721.png' -> './Screenshot 2023-04-21 at 5.20.19 PM-20230423120721.png'  [2 of 10]
42789 of 42789   100% in    0s   432.86 KB/s  done

Removing Files

Deleting a file in S3 is a straightforward process. Deleted files cannot be retrieved.

Remove Command

Remove files

s3cmd rm s3://[bucketname]/filename

s3cmd rm s3://bucketac4/ac-5.txt
delete: 's3://bucketac4/ac-5.txt'

Remove multiple files

s3cmd rm s3://bucketac4/ac-2.txt s3://bucketac4/ac-4.txt
delete: 's3://bucketac4/ac-2.txt'
delete: 's3://bucketac4/ac-4.txt'

Remove all files from a bucket use the recursive and force flag.

s3cmd rm s3://[bucketname]/ --recursive --force

s3cmd get s3://bucketac4/ --recursive
download: 's3://bucketac4/Screenshot 2023-04-18 at 11.03.46 PM-20230423120714.png' -> './Screenshot 2023-04-18 at 11.03.46 PM-20230423120714.png'  [1 of 10]
 512226 of 512226   100% in    0s  1229.82 KB/s  done
download: 's3://bucketac4/Screenshot 2023-04-21 at 5.20.19 PM-20230423120721.png' -> './Screenshot 2023-04-21 at 5.20.19 PM-20230423120721.png'  [2 of 10]
 42789 of 42789   100% in    0s   432.86 KB/s  done

s3cmd (Simple Storage Service Command Line Tool and API)

Install s3cmd

Install

Configure s3cmd

URL and API Keys

Endpoint URL

API Keys

Configure Using s3 Wizard

Start the Wizard

Key Placement

Endpoint URL

DNS

Encryption Password

Path to GPG

HTTPS

Proxy

Validate and Test

Save

Retry

Add Buckets

Make Bucket Command

Removing a Bucket

Remove Bucket Command

List Buckets and Files

List command

Add Files

PUT Command

Retrieving Files

GET Command

Single file download

Multiple file download

Change file name

Use of --recursive. To pull all files from a bucket use the recursive flag.

Removing Files

Remove Command

Remove files

Remove multiple files

Remove all files from a bucket use the recursive and force flag.