Naming Constraints

Object storage types vary in their support for characters and length of file and path names. Avoid transfer problems by correctly naming storage elements.

Path names in Hadoop Distributed File System (HDFS)

By default, each component of a path is limited to 255 bytes in UTF-8 encoding. This value can be configured in the Hadoop configuration file (/etc/hadoop/conf/hdfs-default.xml) by changing the value of dfs.namenode.fs-limits.max-component-length. A value of 0 disables the limit but may create incompatibilities with other file systems that do not support long paths.

Bucket and Container Names

Many object storage platforms (including Amazon S3, Google Cloud Storage, Alibaba Cloud, and Azure, require DNS-compliant bucket names, with additional constraints specific to certain platforms.

Rules for DNS-compliance:

  • Names must be between three and 63 characters long.
  • Names must be a series of one or more labels, with adjacent labels separated by a period (.).
  • Labels can contain lowercase letters, numbers, and hyphens (-), but must start and end with a lowercase letter or a number (labels cannot start or end with a period). Periods may not be adjacent to another period or a hyphen and nor can a hyphen be adjacent to another hyphen. For example, "..", "--", "-.", and ".-" are not valid.
  • Labels cannot be formatted as IP addresses (for example, 192.00.00.20).

Additional Information:

Object Storage Platform Additional Information on Bucket Names
Amazon S3 For more information, see:

http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
Google Cloud Storage
  • Names containing periods may be up to 222 characters total, but each label must be no more than 63 characters.
  • Names may not begin with "goog", nor contain "google" or close misspellings of "google."

For more information, see:

https://cloud.google.com/storage/docs/naming#requirements
Azure For more information, see:

https://docs.microsoft.com/en-us/rest/api/storageservices/fileservices/Naming-and-Referencing-Containers--Blobs--and-Metadata?redirectedfrom=MSDN

Object Names, Key Names, and Blob Names

In general, object names, key names, and blob names must be a sequence of Unicode characters whose UTF-8 encoding is one to 1024 bytes long. This format applies to Amazon S3, Google Cloud Storage, and Azure.

The following character sets are generally safe:

  • Alphanumeric characters: 0-9, a-z, A-Z
  • ! - _ . * ' ( )

The following characters may require special handling, such as URL encoding or referencing as HEX:

  • & $ @ = ; : + , ?
  • spaces
  • ASCII character ranges 00-1F hex (0-31 decimal) and 7F (127 decimal).

Avoid the following characters:

  • \ { } ^ % ` [ ] " < ~ # |
  • Non-printable ASCII characters (128-255 decimal characters)
Object Storage Platform Additional Information on Object Name, Key Name, or Blob Name Requirements
Amazon S3 For more information, see:

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-keys
Google Cloud Storage
  • Names cannot contain Carriage Return or Line Feed characters
  • Avoid control characters that are illegal in XML 1.0 (#x7F -#x84 and #x86-#x9F)

For more information, see:

https://cloud.google.com/storage/docs/naming#objectnames
Azure
  • Blob names are case sensitive.
  • Avoid blob names that end with a period, a forward slash (/), or a sequence of the two.
  • Blob names cannot contain more than 254 path segments, where a path segment is the string between delimiter characters (such as the forward slash) that correspond to the name of a virtual directory.

For more information, see:

https://docs.microsoft.com/en-us/rest/api/storageservices/fileservices/naming-and-referencing-containers--blobs--and-metadata

Object Metadata Names (Keys) and Values

Object metadata is a set of name-value pairs. Users can often add customized metadata names, within the constraints of the object storage platform.

Object Storage Platform Object Metadata Name Requirements
Amazon S3
  • Name-value pairs must conform to US-ASCII when using REST, and to UTF-8 when using SOAP or browser-based uploads (POST requests).
  • When using the REST API, user-defined metadata names must begin with "x-amz-meta-".
  • PUT request headers are limited to 8 Kb, of which 2 Kb may be user-defined metadata. User-defined metadata is calculated as the total bytes of the UTF-8 encoded name and value.

For more information, see:

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-metadata
Google Cloud Storage
  • Custom metadata names must begin with "x-goog-meta-".
  • Each individual metadata entry is limited to 32768 bytes, and 512 Kb for the total metadata server.

For more information, see:

https://cloud.google.com/compute/docs/storing-retrieving-metadata
Azure
  • Metadata names must follow the naming rules for C# identifiers.
  • The combined size of the name-value pair may not exceed 8 Kb.

For more information, see:

https://docs.microsoft.com/en-us/rest/api/storageservices/fileservices/naming-and-referencing-containers--blobs--and-metadata