Naming Constraints
Path names in Hadoop Distributed File System (HDFS)
By default, each component of a path is limited to 255 bytes in UTF-8 encoding. This value can
be configured in the Hadoop configuration file
(/etc/hadoop/conf/hdfs-default.xml) by changing the value of
dfs.namenode.fs-limits.max-component-length
. A value of 0 disables the limit
but may create incompatibilities with other file systems that do not support long paths.
Bucket and Container Names
Many object storage platforms (including AWS S3, Google Cloud Storage, and Azure, require DNS-compliant bucket names, with additional constraints specific to certain platforms. Container names in object storage using the OpenStack Swift API, do not need to be DNS-compliant, as described below.
Rules for DNS-compliance:
- Names must be between three and 63 characters long.
- Names must be a series of one or more labels, with adjacent labels separated by a period (.).
- Labels can contain lowercase letters, numbers, and hyphens (-), but must start and end with a lowercase letter or a number (labels cannot start or end with a period). Periods may not be adjacent to another period or a hyphen and nor can a hyphen be adjacent to another hyphen. For example, "..", "--", "-.", and ".-" are not valid.
- Labels cannot be formatted as IP addresses (for example, 192.00.00.20).
Additional Information:
Object Storage Platform | Additional Information on Bucket Names |
---|---|
AWS S3 | For more information, see: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html |
Google Cloud Storage |
For more information, see: |
Azure | For more information, see: |
Container Names in OpenStack Swift:
Container names must be unique within each account and consist of one to 256 UTF-8 characters. Names can start with any character and contain any character except forward slash (/). For more information, see:
Object Names, Key Names, and Blob Names
In general, object names, key names, and blob names must be a sequence of Unicode characters whose UTF-8 encoding is one to 1024 bytes long. This format applies to AWS S3, Google Cloud Storage, and Azure. Object storage using the OpenStack Swift API has no restrictions on object names.
The following character sets are generally safe:
- Alphanumeric characters: 0-9, a-z, A-Z
- ! - _ . * ' ( )
The following characters may require special handling, such as URL encoding or referencing as HEX:
- & $ @ = ; : + , ?
- spaces
- ASCII character ranges 00-1F hex (0-31 decimal) and 7F (127 decimal).
Avoid the following characters:
- \ { } ^ % ` [ ] " < ~ # |
- Non-printable ASCII characters (128-255 decimal characters)
Object Storage Platform | Additional Information on Object Name, Key Name, or Blob Name Requirements |
---|---|
AWS S3 | For more information, see: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-keys |
Google Cloud Storage |
For more information, see: |
Azure |
For more information, see: |
Object Metadata Names (Keys) and Values
Object metadata is a set of name-value pairs. Users can often add customized metadata names, within the constraints of the object storage platform.
Object Storage Platform | Object Metadata Name Requirements |
---|---|
AWS S3 |
For more information, see: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-metadata |
Google Cloud Storage |
For more information, see: https://cloud.google.com/compute/docs/storing-retrieving-metadata |
OpenStack Swift |
For more information, see: http://developer.openstack.org/api-ref/object-storage/index.html |
Azure |
For more information, see: |