Security and Authentication

What is Active Directory?

To enable read and write access to directories and files for the users on the IBM Spectrum Scale system, you must configure user authentication on the system. Only one user authentication method, and only one instance of that method, can be supported.

The following authentication services can be configured with the IBM Spectrum Scale system for file protocol access:

  • Microsoft Active Directory (AD)
  • Lightweight Directory Access Protocol (LDAP)
  • Network Information Service (NIS) for NFS client access
  • User defined

Active Directory (AD) is a directory service that Microsoft developed for Windows domain networks and is included in most Windows Server operating systems as a set of processes and services. An AD domain controller authenticates and authorizes all users and computers in a Windows domain type network—assigning and enforcing security policies for all computers and installing or updating software.

What is Active Directory Domain?

An Active Directory domain is a collection of objects within a Microsoft Active Directory network. An object can be a single user or a group or it can be a hardware component, such as a computer or printer. Each domain holds a database containing object identity information.

'mmuserauth service create' man page:
  Usage:
  mmuserauth service create [-h|--help] --data-access-method {file|object} --type {ldap|local|ad|nis|userdefined} --servers [--base-dn]
    {[--enable-anonymous-bind]|[--user-name] [--password]} [--enable-server-tls] [--enable-ks-ssl]
    [--enable-kerberos] [--enable-nfs-kerberos] [--enable-ks-casigning]
    [--user-dn] [--group-dn] [--netgroup-dn]
    [--netbios-name] [--domain] [--idmap-role {master|subordinate}] [--idmap-range] [--idmap-range-size]
    [--user-objectclass] [--group-objectclass]
    [--user-name-attrib] [--user-id-attrib] [--user-mail-attrib] [--user-filter]
    [--ks-dns-name] [--ks-admin-user] [--ks-admin-pwd] [--ks-swift-user] [--ks-swift-pwd] [--ks-ext-endpoint]
    [--kerberos-server] [--kerberos-realm]
    [--unixmap-domains]

Configuring IBM Spectrum Scale Object Auth (Keystone) with AD Domain Name for High Availability without TLS

Pre-requisites

* Make sure all the Active Directory Servers have the same domain name.
(For example two Active Directory Servers windowsad1 with IP '10.0.100.1' and windowsad2 with IP '10.0.100.2' share the same domain name 'SPICA.com')

* Configure Specturm Scale Cluster using domian name.
Example: /usr/lpp/mmfs/bin/mmuserauth service create --data-access-method object --type ad --servers 'SPICA.COM' --base-dn 'dc=spica,dc=com' --user-dn 'CN=Users,DC=SPICA,DC=COM' --ks-admin-user 'administrator' --ks-swift-user 'administrator' --ks-swift-pwd 'Passw0rd' --user-name
administrator@spica.com --password Passw0rd

Testing High Availability Senarios without TLS

1) When all the Active Directory Servers sharing the common domain name are running and available:
All the functionality should work as expected.


[root@testnode3 ~]# openstack user list
+---------------+---------------+
| ID            | Name          |
+---------------+---------------+
| Guest         | Guest         |
| krbtgt        | krbtgt        |
| Administrator | Administrator |
| spicauser1    | spicauser1    |
| spicauser2    | spicauser2    |
| spicauser3    | spicauser3    |
+---------------+---------------+

[root@testnode3 ~]# curl -i   -H "Content-Type: application/json"   -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "administrator",
          "domain": { "id": "default" },
          "password": "Passw0rd"
        }
      }
    },
    "scope": {
      "project": {
        "name": "admin",
        "domain": { "id": "default" }
      }
    }
  }
}'   http://127.0.0.1:35357/v3/auth/tokens ; echo
HTTP/1.1 201 Created
Date: Mon, 16 Nov 2015 12:13:35 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
X-Subject-Token: MIIHwgYJKoZIhvcNAQcCoIIHszCCB68CAQExDTALBglghkgBZQMEAgEwggYQBgkqhkiG9w0BBwGgggYBBIIF-XsidG9rZW4iOnsibWV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiI0YWRhYjkzNTkxOGE0ZTlkYjFiYzk5ZmRjOWI0MTA2ZSIsIm5hbWUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTItMTZUMTI6MTM6MzcuMDgzMTUzWiIsInByb2plY3QiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiODQxNmNhYjhiN2I5NDJiN2JmNzYwMjZmYmU3MmNlYzkiLCJuYW1lIjoiYWRtaW4ifSwiY2F0YWxvZyI6W3siZW5kcG9pbnRzIjpbeyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjY4YmJkMWYwZWE4NzQ5ZTRiODIxNjgzYzczNDM1YjNiIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjUyN2ZlNTUwNjM4ODQ0MDZiNjRkZTBmYjU3ZDU2NDQxIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo1MDAwL3YzIiwicmVnaW9uIjpudWxsLCJpbnRlcmZhY2UiOiJwdWJsaWMiLCJpZCI6IjE4NzdkYWM5ZjBkYjRkYzRhYzZjZjA0OWFiOTAzMWU1In1dLCJ0eXBlIjoiaWRlbnRpdHkiLCJpZCI6IjQ2ZDMyZjVhOWZmYjQyMGVhOTg2OGMyYTcxNzA2YzkxIiwibmFtZSI6ImtleXN0b25lIn0seyJlbmRwb2ludHMiOlt7InJlZ2lvbl9pZCI6IlJlZ2lvbk9uZSIsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo4MDgwIiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjE1MzJlNzhiZmRkZTQ3ZWJhNjdmZDk0NjNlNDBlNjY5In0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjY1ZDczYTk0ZDBjZDQwYzRiZjE0MTc0M2ZhNzE1MDVkIn0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoicHVibGljIiwiaWQiOiI2NTE2YzgyNWUzNzA0MGU2OWQ5M2I4OTI4ZjEwNWYxOSJ9XSwidHlwZSI6Im9iamVjdC1zdG9yZSIsImlkIjoiNzQyYWQ0M2U5N2RiNGVmNWIwMGM0NDJjYzBmOGIxNDgiLCJuYW1lIjoic3dpZnQifV0sImV4dHJhcyI6e30sInVzZXIiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiQWRtaW5pc3RyYXRvciIsIm5hbWUiOiJBZG1pbmlzdHJhdG9yIn0sImF1ZGl0X2lkcyI6WyJCV2dyZHotWlJlZUF0Y2pZT0wxSVV3Il0sImlzc3VlZF9hdCI6IjIwMTUtMTEtMTZUMTI6MTM6MzcuMDgzMjAxWiJ9fTGCAYUwggGBAgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBATALBglghkgBZQMEAgEwDQYJKoZIhvcNAQEBBQAEggEAgZo+m91Fu2v9Yj791tWlCGoe4n5mDr2k-nF3d23kz1q701rpMKnmy07M1opSg-KXIYfUmVdx72Fnwy70F8VsfXDKRlRNK0bLT6wGgHOOna1l-Y59A2Tb3EmbbQ7n6EJOuGhtUtqmt+KkLS8cl+heU9L3sZ4590AK-bm57H0eG7wKzURmDLkrAWhlgBc029F-w8Jd13ncVjhs+rY0vaMEHEvDlQrnI-We9pW3HzNFS66UYB2Ut7BmMpdjIY6LKniJmw2SzjfP2kO2nLz9coLDLx-UJcpvXSW6mYOZSRrcOOTEIpUgAPigqWCSsFECHvvl+e0CEo08pqqrxgx60KBKAw==
Vary: X-Auth-Token
x-openstack-request-id: req-482f63c2-9517-466c-b208-136b3d69ec62
Content-Length: 1643
Content-Type: application/json

{"token": {"methods": ["password"], "roles": [{"id": "4adab935918a4e9db1bc99fdc9b4106e", "name": "admin"}], "expires_at": "2015-12-16T12:13:37.083153Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "8416cab8b7b942b7bf76026fbe72cec9", "name": "admin"}, "catalog": [{"endpoints": [{"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "admin", "id": "68bbd1f0ea8749e4b821683c73435b3b"}, {"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "internal", "id": "527fe55063884406b64de0fb57d56441"}, {"region_id": null, "url": "http://10.0.100.45:5000/v3", "region": null, "interface": "public", "id": "1877dac9f0db4dc4ac6cf049ab9031e5"}], "type": "identity", "id": "46d32f5a9ffb420ea9868c2a71706c91", "name": "keystone"}, {"endpoints": [{"region_id": "RegionOne", "url": "http://10.0.100.45:8080", "region": "RegionOne", "interface": "admin", "id": "1532e78bfdde47eba67fd9463e40e669"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "internal", "id": "65d73a94d0cd40c4bf141743fa71505d"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "public", "id": "6516c825e37040e69d93b8928f105f19"}], "type": "object-store", "id": "742ad43e97db4ef5b00c442cc0f8b148", "name": "swift"}], "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "Administrator", "name": "Administrator"}, "audit_ids": ["BWgrdz-ZReeAtcjYOL1IUw"], "issued_at": "2015-11-16T12:13:37.083201Z"}}
				

2) When one or more Active Directory Servers sharing the common domain name are available and one or more Active Directory Servers sharing the common domain name are down :
All the functionality should work as expected.
Note: Expect some delay in response


[root@testnode3 ~]# openstack user list
+---------------+---------------+
| ID            | Name          |
+---------------+---------------+
| Guest         | Guest         |
| krbtgt        | krbtgt        |
| Administrator | Administrator |
| spicauser1    | spicauser1    |
| spicauser2    | spicauser2    |
| spicauser3    | spicauser3    |
+---------------+---------------+

[root@testnode3 ~]# curl -i   -H "Content-Type: application/json"   -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "administrator",
          "domain": { "id": "default" },
          "password": "Passw0rd"
        }
      }
    },
    "scope": {
      "project": {
        "name": "admin",
        "domain": { "id": "default" }
      }
    }
  }
}'   http://127.0.0.1:35357/v3/auth/tokens ; echo
HTTP/1.1 201 Created
Date: Mon, 16 Nov 2015 12:13:35 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
X-Subject-Token: MIIHwgYJKoZIhvcNAQcCoIIHszCCB68CAQExDTALBglghkgBZQMEAgEwggYQBgkqhkiG9w0BBwGgggYBBIIF-XsidG9rZW4iOnsibWV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiI0YWRhYjkzNTkxOGE0ZTlkYjFiYzk5ZmRjOWI0MTA2ZSIsIm5hbWUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTItMTZUMTI6MTM6MzcuMDgzMTUzWiIsInByb2plY3QiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiODQxNmNhYjhiN2I5NDJiN2JmNzYwMjZmYmU3MmNlYzkiLCJuYW1lIjoiYWRtaW4ifSwiY2F0YWxvZyI6W3siZW5kcG9pbnRzIjpbeyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjY4YmJkMWYwZWE4NzQ5ZTRiODIxNjgzYzczNDM1YjNiIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjUyN2ZlNTUwNjM4ODQ0MDZiNjRkZTBmYjU3ZDU2NDQxIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo1MDAwL3YzIiwicmVnaW9uIjpudWxsLCJpbnRlcmZhY2UiOiJwdWJsaWMiLCJpZCI6IjE4NzdkYWM5ZjBkYjRkYzRhYzZjZjA0OWFiOTAzMWU1In1dLCJ0eXBlIjoiaWRlbnRpdHkiLCJpZCI6IjQ2ZDMyZjVhOWZmYjQyMGVhOTg2OGMyYTcxNzA2YzkxIiwibmFtZSI6ImtleXN0b25lIn0seyJlbmRwb2ludHMiOlt7InJlZ2lvbl9pZCI6IlJlZ2lvbk9uZSIsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo4MDgwIiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjE1MzJlNzhiZmRkZTQ3ZWJhNjdmZDk0NjNlNDBlNjY5In0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjY1ZDczYTk0ZDBjZDQwYzRiZjE0MTc0M2ZhNzE1MDVkIn0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoicHVibGljIiwiaWQiOiI2NTE2YzgyNWUzNzA0MGU2OWQ5M2I4OTI4ZjEwNWYxOSJ9XSwidHlwZSI6Im9iamVjdC1zdG9yZSIsImlkIjoiNzQyYWQ0M2U5N2RiNGVmNWIwMGM0NDJjYzBmOGIxNDgiLCJuYW1lIjoic3dpZnQifV0sImV4dHJhcyI6e30sInVzZXIiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiQWRtaW5pc3RyYXRvciIsIm5hbWUiOiJBZG1pbmlzdHJhdG9yIn0sImF1ZGl0X2lkcyI6WyJCV2dyZHotWlJlZUF0Y2pZT0wxSVV3Il0sImlzc3VlZF9hdCI6IjIwMTUtMTEtMTZUMTI6MTM6MzcuMDgzMjAxWiJ9fTGCAYUwggGBAgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBATALBglghkgBZQMEAgEwDQYJKoZIhvcNAQEBBQAEggEAgZo+m91Fu2v9Yj791tWlCGoe4n5mDr2k-nF3d23kz1q701rpMKnmy07M1opSg-KXIYfUmVdx72Fnwy70F8VsfXDKRlRNK0bLT6wGgHOOna1l-Y59A2Tb3EmbbQ7n6EJOuGhtUtqmt+KkLS8cl+heU9L3sZ4590AK-bm57H0eG7wKzURmDLkrAWhlgBc029F-w8Jd13ncVjhs+rY0vaMEHEvDlQrnI-We9pW3HzNFS66UYB2Ut7BmMpdjIY6LKniJmw2SzjfP2kO2nLz9coLDLx-UJcpvXSW6mYOZSRrcOOTEIpUgAPigqWCSsFECHvvl+e0CEo08pqqrxgx60KBKAw==
Vary: X-Auth-Token
x-openstack-request-id: req-482f63c2-9517-466c-b208-136b3d69ec62
Content-Length: 1643
Content-Type: application/json

{"token": {"methods": ["password"], "roles": [{"id": "4adab935918a4e9db1bc99fdc9b4106e", "name": "admin"}], "expires_at": "2015-12-16T12:13:37.083153Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "8416cab8b7b942b7bf76026fbe72cec9", "name": "admin"}, "catalog": [{"endpoints": [{"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "admin", "id": "68bbd1f0ea8749e4b821683c73435b3b"}, {"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "internal", "id": "527fe55063884406b64de0fb57d56441"}, {"region_id": null, "url": "http://10.0.100.45:5000/v3", "region": null, "interface": "public", "id": "1877dac9f0db4dc4ac6cf049ab9031e5"}], "type": "identity", "id": "46d32f5a9ffb420ea9868c2a71706c91", "name": "keystone"}, {"endpoints": [{"region_id": "RegionOne", "url": "http://10.0.100.45:8080", "region": "RegionOne", "interface": "admin", "id": "1532e78bfdde47eba67fd9463e40e669"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "internal", "id": "65d73a94d0cd40c4bf141743fa71505d"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "public", "id": "6516c825e37040e69d93b8928f105f19"}], "type": "object-store", "id": "742ad43e97db4ef5b00c442cc0f8b148", "name": "swift"}], "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "Administrator", "name": "Administrator"}, "audit_ids": ["BWgrdz-ZReeAtcjYOL1IUw"], "issued_at": "2015-11-16T12:13:37.083201Z"}}
				

3) When all the Active Directory Servers sharing the common domain name are un-available :
Then return code of 500 is expected.
Note: Expect some delay in response


[root@testnode3 ~]# openstack user list
ERROR: openstack An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-6e66e336-3492-44c7-8a55-9070ac504127

[root@testnode3 ~]# curl -i   -H "Content-Type: application/json"   -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "administrator",
          "domain": { "id": "default" },
          "password": "Passw0rd"
        }
      }
    },
    "scope": {
      "project": {
        "name": "admin",
        "domain": { "id": "default" }
      }
    }
  }
}'   http://127.0.0.1:35357/v3/auth/tokens ; echo
HTTP/1.1 201 Created
Date: Mon, 16 Nov 2015 12:13:35 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
X-Subject-Token: MIIHwgYJKoZIhvcNAQcCoIIHszCCB68CAQExDTALBglghkgBZQMEAgEwggYQBgkqhkiG9w0BBwGgggYBBIIF-XsidG9rZW4iOnsibWV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiI0YWRhYjkzNTkxOGE0ZTlkYjFiYzk5ZmRjOWI0MTA2ZSIsIm5hbWUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTItMTZUMTI6MTM6MzcuMDgzMTUzWiIsInByb2plY3QiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiODQxNmNhYjhiN2I5NDJiN2JmNzYwMjZmYmU3MmNlYzkiLCJuYW1lIjoiYWRtaW4ifSwiY2F0YWxvZyI6W3siZW5kcG9pbnRzIjpbeyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjY4YmJkMWYwZWE4NzQ5ZTRiODIxNjgzYzczNDM1YjNiIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjUyN2ZlNTUwNjM4ODQ0MDZiNjRkZTBmYjU3ZDU2NDQxIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo1MDAwL3YzIiwicmVnaW9uIjpudWxsLCJpbnRlcmZhY2UiOiJwdWJsaWMiLCJpZCI6IjE4NzdkYWM5ZjBkYjRkYzRhYzZjZjA0OWFiOTAzMWU1In1dLCJ0eXBlIjoiaWRlbnRpdHkiLCJpZCI6IjQ2ZDMyZjVhOWZmYjQyMGVhOTg2OGMyYTcxNzA2YzkxIiwibmFtZSI6ImtleXN0b25lIn0seyJlbmRwb2ludHMiOlt7InJlZ2lvbl9pZCI6IlJlZ2lvbk9uZSIsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo4MDgwIiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjE1MzJlNzhiZmRkZTQ3ZWJhNjdmZDk0NjNlNDBlNjY5In0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjY1ZDczYTk0ZDBjZDQwYzRiZjE0MTc0M2ZhNzE1MDVkIn0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoicHVibGljIiwiaWQiOiI2NTE2YzgyNWUzNzA0MGU2OWQ5M2I4OTI4ZjEwNWYxOSJ9XSwidHlwZSI6Im9iamVjdC1zdG9yZSIsImlkIjoiNzQyYWQ0M2U5N2RiNGVmNWIwMGM0NDJjYzBmOGIxNDgiLCJuYW1lIjoic3dpZnQifV0sImV4dHJhcyI6e30sInVzZXIiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiQWRtaW5pc3RyYXRvciIsIm5hbWUiOiJBZG1pbmlzdHJhdG9yIn0sImF1ZGl0X2lkcyI6WyJCV2dyZHotWlJlZUF0Y2pZT0wxSVV3Il0sImlzc3VlZF9hdCI6IjIwMTUtMTEtMTZUMTI6MTM6MzcuMDgzMjAxWiJ9fTGCAYUwggGBAgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBATALBglghkgBZQMEAgEwDQYJKoZIhvcNAQEBBQAEggEAgZo+m91Fu2v9Yj791tWlCGoe4n5mDr2k-nF3d23kz1q701rpMKnmy07M1opSg-KXIYfUmVdx72Fnwy70F8VsfXDKRlRNK0bLT6wGgHOOna1l-Y59A2Tb3EmbbQ7n6EJOuGhtUtqmt+KkLS8cl+heU9L3sZ4590AK-bm57H0eG7wKzURmDLkrAWhlgBc029F-w8Jd13ncVjhs+rY0vaMEHEvDlQrnI-We9pW3HzNFS66UYB2Ut7BmMpdjIY6LKniJmw2SzjfP2kO2nLz9coLDLx-UJcpvXSW6mYOZSRrcOOTEIpUgAPigqWCSsFECHvvl+e0CEo08pqqrxgx60KBKAw==
Vary: X-Auth-Token
x-openstack-request-id: req-482f63c2-9517-466c-b208-136b3d69ec62
Content-Length: 1643
Content-Type: application/json

{"token": {"methods": ["password"], "roles": [{"id": "4adab935918a4e9db1bc99fdc9b4106e", "name": "admin"}], "expires_at": "2015-12-16T12:13:37.083153Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "8416cab8b7b942b7bf76026fbe72cec9", "name": "admin"}, "catalog": [{"endpoints": [{"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "admin", "id": "68bbd1f0ea8749e4b821683c73435b3b"}, {"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "internal", "id": "527fe55063884406b64de0fb57d56441"}, {"region_id": null, "url": "http://10.0.100.45:5000/v3", "region": null, "interface": "public", "id": "1877dac9f0db4dc4ac6cf049ab9031e5"}], "type": "identity", "id": "46d32f5a9ffb420ea9868c2a71706c91", "name": "keystone"}, {"endpoints": [{"region_id": "RegionOne", "url": "http://10.0.100.45:8080", "region": "RegionOne", "interface": "admin", "id": "1532e78bfdde47eba67fd9463e40e669"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "internal", "id": "65d73a94d0cd40c4bf141743fa71505d"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "public", "id": "6516c825e37040e69d93b8928f105f19"}], "type": "object-store", "id": "742ad43e97db4ef5b00c442cc0f8b148", "name": "swift"}], "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "Administrator", "name": "Administrator"}, "audit_ids": ["BWgrdz-ZReeAtcjYOL1IUw"], "issued_at": "2015-11-16T12:13:37.083201Z"}}
				


Configuring IBM Spectrum Scale Object Auth (Keystone) with AD Domain Name for High Availability with TLS

Pre-requisites

* Make sure all the Active Directory Servers have the same domain name.
(For example two Active Directory Servers windowsad1 with IP '10.0.100.1' and windowsad2 with IP '10.0.100.2' share the same domain name 'SPICA.com')

* Tls certificate : Used by keystone for connecting to ldap|ad server over tls. If user specifies the flag --enable-server-tls then mmuserauth expect following certificate file to be present on current node. Valid with --data-access-method object and --type ad|ldap /var/mmfs/tmp/object_ldap_cacert.pem

* Refer to the following link for creating a certificate. http://viralmutant.blogspot.in/2015/06/put-that-wretched-ca-to-some-use.html#.Vksa-tSSxPE * Configure Specturm Scale Cluster using domian name and TLS.
Example:


[root@testnode3 tmp]# mv /var/mmfs/tmp/object_ldap_cacert.pem /var/mmfs/tmp/object_ldap_cacert.pem.bk

[root@testnode3 tmp]# /usr/lpp/mmfs/bin/mmuserauth service create --data-access-method object --type ad --servers 'SPICA.COM' --base-dn 'dc=spica,dc=com' --user-dn 'CN=Users,DC=SPICA,DC=COM' --ks-admin-user 'administrator' --ks-swift-user 'administrator' --ks-swift-pwd 'Passw0rd' --user-name administrator@spica.com --password Passw0rd --enable-server-tls
/var/mmfs/tmp/object_ldap_cacert.pem: [E] File not found at specified location
mmuserauth service create: Command failed. Examine previous error messages to determine cause.

[root@testnode3 tmp]# mv /tmp/object_ldap_cacert.pem /var/mmfs/tmp/object_ldap_cacert.pem

[root@testnode3 tmp]# stat /var/mmfs/tmp/object_ldap_cacert.pem  File: ‘/var/mmfs/tmp/object_ldap_cacert.pem’
  Size: 1946      	Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d	Inode: 69364021    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: unconfined_u:object_r:user_tmp_t:s0
Access: 2015-11-19 14:14:52.068000000 +0530
Modify: 2015-11-19 14:14:52.068000000 +0530
Change: 2015-11-19 14:38:29.593000000 +0530
 Birth: -

[root@testnode3 tmp]# /usr/lpp/mmfs/bin/mmuserauth service create --data-access-method object --type ad --servers 'SPICA.COM' --base-dn 'dc=spica,dc=com' --user-dn 'CN=Users,DC=SPICA,DC=COM' --ks-admin-user 'administrator' --ks-swift-user 'administrator' --ks-swift-pwd 'Passw0rd' --user-name administrator@spica.com --password Passw0rd --enable-server-tls
mmcesobjcrbase: Validating execution environment.
mmcesobjcrbase: Performing SELinux configuration.
mmcesobjcrbase: Configuring Keystone server in /ibm/gpfs0/ces/object/keystone.
mmcesobjcrbase: Initiating action (start) on postgres in the cluster.
mmcesobjcrbase: Validating Keystone environment.
mmcesobjcrbase: Validating Swift values in Keystone.
mmcesobjcrbase: Configuration complete.
Object configuration with LDAP (Active Directory) as the identity backend has completed successfully.
Object authentication configuration completed successfully.

[root@testnode3 ~]# openstack user list
ERROR: openstack An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-330d47f1-7b46-4de0-bf0f-097b909ed85f)
				

Note: If the above error is encountered, follow the below workaround, to make sure TLS certificate is properly copied to all protocol nodes.

[root@testnode3 ~]# mmuserauth service check --data-access-method object -N cesNodes -r
Userauth object check on node: testnode3
	Checking keystone.conf: OK
	Checking wsgi-keystone.conf: OK
	Checking /etc/keystone/ssl/certs/signing_cert.pem: OK
	Checking /etc/keystone/ssl/private/signing_key.pem: OK
	Checking /etc/keystone/ssl/certs/signing_cacert.pem: OK
	Checking /etc/keystone/ssl/certs/object_ldap_cacert.pem: OK
Service 'httpd' status: OK

Userauth object check on node: testnode4
	Checking keystone.conf: OK
	Checking wsgi-keystone.conf: OK
	Checking /etc/keystone/ssl/certs/signing_cert.pem: OK
	Checking /etc/keystone/ssl/private/signing_key.pem: OK
	Checking /etc/keystone/ssl/certs/signing_cacert.pem: OK
	Checking /etc/keystone/ssl/certs/object_ldap_cacert.pem: Missing
Restored the missing certificate files
Configuration Corrected. Restarting daemons.
Service 'httpd' status: OK
Restarting httpd
				

Testing High Availability Senarios with TLS

1) When all the Active Directory Servers sharing the common domain name are running and available:
All the functionality should work as expected.


[root@testnode3 ~]# openstack user list
+---------------+---------------+
| ID            | Name          |
+---------------+---------------+
| Guest         | Guest         |
| krbtgt        | krbtgt        |
| Administrator | Administrator |
| spicauser1    | spicauser1    |
| spicauser2    | spicauser2    |
| spicauser3    | spicauser3    |
+---------------+---------------+

[root@testnode3 ~]# curl -i   -H "Content-Type: application/json"   -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "administrator",
          "domain": { "id": "default" },
          "password": "Passw0rd"
        }
      }
    },
    "scope": {
      "project": {
        "name": "admin",
        "domain": { "id": "default" }
      }
    }
  }
}'   http://127.0.0.1:35357/v3/auth/tokens ; echo
HTTP/1.1 201 Created
Date: Mon, 16 Nov 2015 12:13:35 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
X-Subject-Token: MIIHwgYJKoZIhvcNAQcCoIIHszCCB68CAQExDTALBglghkgBZQMEAgEwggYQBgkqhkiG9w0BBwGgggYBBIIF-XsidG9rZW4iOnsibWV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiI0YWRhYjkzNTkxOGE0ZTlkYjFiYzk5ZmRjOWI0MTA2ZSIsIm5hbWUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTItMTZUMTI6MTM6MzcuMDgzMTUzWiIsInByb2plY3QiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiODQxNmNhYjhiN2I5NDJiN2JmNzYwMjZmYmU3MmNlYzkiLCJuYW1lIjoiYWRtaW4ifSwiY2F0YWxvZyI6W3siZW5kcG9pbnRzIjpbeyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjY4YmJkMWYwZWE4NzQ5ZTRiODIxNjgzYzczNDM1YjNiIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjUyN2ZlNTUwNjM4ODQ0MDZiNjRkZTBmYjU3ZDU2NDQxIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo1MDAwL3YzIiwicmVnaW9uIjpudWxsLCJpbnRlcmZhY2UiOiJwdWJsaWMiLCJpZCI6IjE4NzdkYWM5ZjBkYjRkYzRhYzZjZjA0OWFiOTAzMWU1In1dLCJ0eXBlIjoiaWRlbnRpdHkiLCJpZCI6IjQ2ZDMyZjVhOWZmYjQyMGVhOTg2OGMyYTcxNzA2YzkxIiwibmFtZSI6ImtleXN0b25lIn0seyJlbmRwb2ludHMiOlt7InJlZ2lvbl9pZCI6IlJlZ2lvbk9uZSIsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo4MDgwIiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjE1MzJlNzhiZmRkZTQ3ZWJhNjdmZDk0NjNlNDBlNjY5In0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjY1ZDczYTk0ZDBjZDQwYzRiZjE0MTc0M2ZhNzE1MDVkIn0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoicHVibGljIiwiaWQiOiI2NTE2YzgyNWUzNzA0MGU2OWQ5M2I4OTI4ZjEwNWYxOSJ9XSwidHlwZSI6Im9iamVjdC1zdG9yZSIsImlkIjoiNzQyYWQ0M2U5N2RiNGVmNWIwMGM0NDJjYzBmOGIxNDgiLCJuYW1lIjoic3dpZnQifV0sImV4dHJhcyI6e30sInVzZXIiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiQWRtaW5pc3RyYXRvciIsIm5hbWUiOiJBZG1pbmlzdHJhdG9yIn0sImF1ZGl0X2lkcyI6WyJCV2dyZHotWlJlZUF0Y2pZT0wxSVV3Il0sImlzc3VlZF9hdCI6IjIwMTUtMTEtMTZUMTI6MTM6MzcuMDgzMjAxWiJ9fTGCAYUwggGBAgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBATALBglghkgBZQMEAgEwDQYJKoZIhvcNAQEBBQAEggEAgZo+m91Fu2v9Yj791tWlCGoe4n5mDr2k-nF3d23kz1q701rpMKnmy07M1opSg-KXIYfUmVdx72Fnwy70F8VsfXDKRlRNK0bLT6wGgHOOna1l-Y59A2Tb3EmbbQ7n6EJOuGhtUtqmt+KkLS8cl+heU9L3sZ4590AK-bm57H0eG7wKzURmDLkrAWhlgBc029F-w8Jd13ncVjhs+rY0vaMEHEvDlQrnI-We9pW3HzNFS66UYB2Ut7BmMpdjIY6LKniJmw2SzjfP2kO2nLz9coLDLx-UJcpvXSW6mYOZSRrcOOTEIpUgAPigqWCSsFECHvvl+e0CEo08pqqrxgx60KBKAw==
Vary: X-Auth-Token
x-openstack-request-id: req-482f63c2-9517-466c-b208-136b3d69ec62
Content-Length: 1643
Content-Type: application/json

{"token": {"methods": ["password"], "roles": [{"id": "4adab935918a4e9db1bc99fdc9b4106e", "name": "admin"}], "expires_at": "2015-12-16T12:13:37.083153Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "8416cab8b7b942b7bf76026fbe72cec9", "name": "admin"}, "catalog": [{"endpoints": [{"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "admin", "id": "68bbd1f0ea8749e4b821683c73435b3b"}, {"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "internal", "id": "527fe55063884406b64de0fb57d56441"}, {"region_id": null, "url": "http://10.0.100.45:5000/v3", "region": null, "interface": "public", "id": "1877dac9f0db4dc4ac6cf049ab9031e5"}], "type": "identity", "id": "46d32f5a9ffb420ea9868c2a71706c91", "name": "keystone"}, {"endpoints": [{"region_id": "RegionOne", "url": "http://10.0.100.45:8080", "region": "RegionOne", "interface": "admin", "id": "1532e78bfdde47eba67fd9463e40e669"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "internal", "id": "65d73a94d0cd40c4bf141743fa71505d"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "public", "id": "6516c825e37040e69d93b8928f105f19"}], "type": "object-store", "id": "742ad43e97db4ef5b00c442cc0f8b148", "name": "swift"}], "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "Administrator", "name": "Administrator"}, "audit_ids": ["BWgrdz-ZReeAtcjYOL1IUw"], "issued_at": "2015-11-16T12:13:37.083201Z"}}
				

2) When one or more Active Directory Servers sharing the common domain name are available and one or more Active Directory Servers sharing the common domain name are down :
All the functionality should work as expected.
Note: Expect some delay in response


[root@testnode3 ~]# openstack user list
+---------------+---------------+
| ID            | Name          |
+---------------+---------------+
| Guest         | Guest         |
| krbtgt        | krbtgt        |
| Administrator | Administrator |
| spicauser1    | spicauser1    |
| spicauser2    | spicauser2    |
| spicauser3    | spicauser3    |
+---------------+---------------+

[root@testnode3 ~]# curl -i   -H "Content-Type: application/json"   -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "administrator",
          "domain": { "id": "default" },
          "password": "Passw0rd"
        }
      }
    },
    "scope": {
      "project": {
        "name": "admin",
        "domain": { "id": "default" }
      }
    }
  }
}'   http://127.0.0.1:35357/v3/auth/tokens ; echo
HTTP/1.1 201 Created
Date: Mon, 16 Nov 2015 12:13:35 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
X-Subject-Token: MIIHwgYJKoZIhvcNAQcCoIIHszCCB68CAQExDTALBglghkgBZQMEAgEwggYQBgkqhkiG9w0BBwGgggYBBIIF-XsidG9rZW4iOnsibWV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiI0YWRhYjkzNTkxOGE0ZTlkYjFiYzk5ZmRjOWI0MTA2ZSIsIm5hbWUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTItMTZUMTI6MTM6MzcuMDgzMTUzWiIsInByb2plY3QiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiODQxNmNhYjhiN2I5NDJiN2JmNzYwMjZmYmU3MmNlYzkiLCJuYW1lIjoiYWRtaW4ifSwiY2F0YWxvZyI6W3siZW5kcG9pbnRzIjpbeyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjY4YmJkMWYwZWE4NzQ5ZTRiODIxNjgzYzczNDM1YjNiIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjUyN2ZlNTUwNjM4ODQ0MDZiNjRkZTBmYjU3ZDU2NDQxIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo1MDAwL3YzIiwicmVnaW9uIjpudWxsLCJpbnRlcmZhY2UiOiJwdWJsaWMiLCJpZCI6IjE4NzdkYWM5ZjBkYjRkYzRhYzZjZjA0OWFiOTAzMWU1In1dLCJ0eXBlIjoiaWRlbnRpdHkiLCJpZCI6IjQ2ZDMyZjVhOWZmYjQyMGVhOTg2OGMyYTcxNzA2YzkxIiwibmFtZSI6ImtleXN0b25lIn0seyJlbmRwb2ludHMiOlt7InJlZ2lvbl9pZCI6IlJlZ2lvbk9uZSIsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo4MDgwIiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjE1MzJlNzhiZmRkZTQ3ZWJhNjdmZDk0NjNlNDBlNjY5In0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjY1ZDczYTk0ZDBjZDQwYzRiZjE0MTc0M2ZhNzE1MDVkIn0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoicHVibGljIiwiaWQiOiI2NTE2YzgyNWUzNzA0MGU2OWQ5M2I4OTI4ZjEwNWYxOSJ9XSwidHlwZSI6Im9iamVjdC1zdG9yZSIsImlkIjoiNzQyYWQ0M2U5N2RiNGVmNWIwMGM0NDJjYzBmOGIxNDgiLCJuYW1lIjoic3dpZnQifV0sImV4dHJhcyI6e30sInVzZXIiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiQWRtaW5pc3RyYXRvciIsIm5hbWUiOiJBZG1pbmlzdHJhdG9yIn0sImF1ZGl0X2lkcyI6WyJCV2dyZHotWlJlZUF0Y2pZT0wxSVV3Il0sImlzc3VlZF9hdCI6IjIwMTUtMTEtMTZUMTI6MTM6MzcuMDgzMjAxWiJ9fTGCAYUwggGBAgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBATALBglghkgBZQMEAgEwDQYJKoZIhvcNAQEBBQAEggEAgZo+m91Fu2v9Yj791tWlCGoe4n5mDr2k-nF3d23kz1q701rpMKnmy07M1opSg-KXIYfUmVdx72Fnwy70F8VsfXDKRlRNK0bLT6wGgHOOna1l-Y59A2Tb3EmbbQ7n6EJOuGhtUtqmt+KkLS8cl+heU9L3sZ4590AK-bm57H0eG7wKzURmDLkrAWhlgBc029F-w8Jd13ncVjhs+rY0vaMEHEvDlQrnI-We9pW3HzNFS66UYB2Ut7BmMpdjIY6LKniJmw2SzjfP2kO2nLz9coLDLx-UJcpvXSW6mYOZSRrcOOTEIpUgAPigqWCSsFECHvvl+e0CEo08pqqrxgx60KBKAw==
Vary: X-Auth-Token
x-openstack-request-id: req-482f63c2-9517-466c-b208-136b3d69ec62
Content-Length: 1643
Content-Type: application/json

{"token": {"methods": ["password"], "roles": [{"id": "4adab935918a4e9db1bc99fdc9b4106e", "name": "admin"}], "expires_at": "2015-12-16T12:13:37.083153Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "8416cab8b7b942b7bf76026fbe72cec9", "name": "admin"}, "catalog": [{"endpoints": [{"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "admin", "id": "68bbd1f0ea8749e4b821683c73435b3b"}, {"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "internal", "id": "527fe55063884406b64de0fb57d56441"}, {"region_id": null, "url": "http://10.0.100.45:5000/v3", "region": null, "interface": "public", "id": "1877dac9f0db4dc4ac6cf049ab9031e5"}], "type": "identity", "id": "46d32f5a9ffb420ea9868c2a71706c91", "name": "keystone"}, {"endpoints": [{"region_id": "RegionOne", "url": "http://10.0.100.45:8080", "region": "RegionOne", "interface": "admin", "id": "1532e78bfdde47eba67fd9463e40e669"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "internal", "id": "65d73a94d0cd40c4bf141743fa71505d"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "public", "id": "6516c825e37040e69d93b8928f105f19"}], "type": "object-store", "id": "742ad43e97db4ef5b00c442cc0f8b148", "name": "swift"}], "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "Administrator", "name": "Administrator"}, "audit_ids": ["BWgrdz-ZReeAtcjYOL1IUw"], "issued_at": "2015-11-16T12:13:37.083201Z"}}
				

3) When all the Active Directory Servers sharing the common domain name are un-available :
Then return code of 500 is expected.
Note: Expect some delay in response


[root@testnode3 ~]# openstack user list
ERROR: openstack An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-6e66e336-3492-44c7-8a55-9070ac504127

[root@testnode3 ~]# curl -i   -H "Content-Type: application/json"   -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "administrator",
          "domain": { "id": "default" },
          "password": "Passw0rd"
        }
      }
    },
    "scope": {
      "project": {
        "name": "admin",
        "domain": { "id": "default" }
      }
    }
  }
}'   http://127.0.0.1:35357/v3/auth/tokens ; echo
HTTP/1.1 201 Created
Date: Mon, 16 Nov 2015 12:13:35 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
X-Subject-Token: MIIHwgYJKoZIhvcNAQcCoIIHszCCB68CAQExDTALBglghkgBZQMEAgEwggYQBgkqhkiG9w0BBwGgggYBBIIF-XsidG9rZW4iOnsibWV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiI0YWRhYjkzNTkxOGE0ZTlkYjFiYzk5ZmRjOWI0MTA2ZSIsIm5hbWUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTItMTZUMTI6MTM6MzcuMDgzMTUzWiIsInByb2plY3QiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiODQxNmNhYjhiN2I5NDJiN2JmNzYwMjZmYmU3MmNlYzkiLCJuYW1lIjoiYWRtaW4ifSwiY2F0YWxvZyI6W3siZW5kcG9pbnRzIjpbeyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjY4YmJkMWYwZWE4NzQ5ZTRiODIxNjgzYzczNDM1YjNiIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTozNTM1Ny92MyIsInJlZ2lvbiI6bnVsbCwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjUyN2ZlNTUwNjM4ODQ0MDZiNjRkZTBmYjU3ZDU2NDQxIn0seyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo1MDAwL3YzIiwicmVnaW9uIjpudWxsLCJpbnRlcmZhY2UiOiJwdWJsaWMiLCJpZCI6IjE4NzdkYWM5ZjBkYjRkYzRhYzZjZjA0OWFiOTAzMWU1In1dLCJ0eXBlIjoiaWRlbnRpdHkiLCJpZCI6IjQ2ZDMyZjVhOWZmYjQyMGVhOTg2OGMyYTcxNzA2YzkxIiwibmFtZSI6ImtleXN0b25lIn0seyJlbmRwb2ludHMiOlt7InJlZ2lvbl9pZCI6IlJlZ2lvbk9uZSIsInVybCI6Imh0dHA6Ly8xMC4wLjEwMC40NTo4MDgwIiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiYWRtaW4iLCJpZCI6IjE1MzJlNzhiZmRkZTQ3ZWJhNjdmZDk0NjNlNDBlNjY5In0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoiaW50ZXJuYWwiLCJpZCI6IjY1ZDczYTk0ZDBjZDQwYzRiZjE0MTc0M2ZhNzE1MDVkIn0seyJyZWdpb25faWQiOiJSZWdpb25PbmUiLCJ1cmwiOiJodHRwOi8vMTAuMC4xMDAuNDU6ODA4MC92MS9BVVRIXzg0MTZjYWI4YjdiOTQyYjdiZjc2MDI2ZmJlNzJjZWM5IiwicmVnaW9uIjoiUmVnaW9uT25lIiwiaW50ZXJmYWNlIjoicHVibGljIiwiaWQiOiI2NTE2YzgyNWUzNzA0MGU2OWQ5M2I4OTI4ZjEwNWYxOSJ9XSwidHlwZSI6Im9iamVjdC1zdG9yZSIsImlkIjoiNzQyYWQ0M2U5N2RiNGVmNWIwMGM0NDJjYzBmOGIxNDgiLCJuYW1lIjoic3dpZnQifV0sImV4dHJhcyI6e30sInVzZXIiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiQWRtaW5pc3RyYXRvciIsIm5hbWUiOiJBZG1pbmlzdHJhdG9yIn0sImF1ZGl0X2lkcyI6WyJCV2dyZHotWlJlZUF0Y2pZT0wxSVV3Il0sImlzc3VlZF9hdCI6IjIwMTUtMTEtMTZUMTI6MTM6MzcuMDgzMjAxWiJ9fTGCAYUwggGBAgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBATALBglghkgBZQMEAgEwDQYJKoZIhvcNAQEBBQAEggEAgZo+m91Fu2v9Yj791tWlCGoe4n5mDr2k-nF3d23kz1q701rpMKnmy07M1opSg-KXIYfUmVdx72Fnwy70F8VsfXDKRlRNK0bLT6wGgHOOna1l-Y59A2Tb3EmbbQ7n6EJOuGhtUtqmt+KkLS8cl+heU9L3sZ4590AK-bm57H0eG7wKzURmDLkrAWhlgBc029F-w8Jd13ncVjhs+rY0vaMEHEvDlQrnI-We9pW3HzNFS66UYB2Ut7BmMpdjIY6LKniJmw2SzjfP2kO2nLz9coLDLx-UJcpvXSW6mYOZSRrcOOTEIpUgAPigqWCSsFECHvvl+e0CEo08pqqrxgx60KBKAw==
Vary: X-Auth-Token
x-openstack-request-id: req-482f63c2-9517-466c-b208-136b3d69ec62
Content-Length: 1643
Content-Type: application/json

{"token": {"methods": ["password"], "roles": [{"id": "4adab935918a4e9db1bc99fdc9b4106e", "name": "admin"}], "expires_at": "2015-12-16T12:13:37.083153Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "8416cab8b7b942b7bf76026fbe72cec9", "name": "admin"}, "catalog": [{"endpoints": [{"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "admin", "id": "68bbd1f0ea8749e4b821683c73435b3b"}, {"region_id": null, "url": "http://10.0.100.45:35357/v3", "region": null, "interface": "internal", "id": "527fe55063884406b64de0fb57d56441"}, {"region_id": null, "url": "http://10.0.100.45:5000/v3", "region": null, "interface": "public", "id": "1877dac9f0db4dc4ac6cf049ab9031e5"}], "type": "identity", "id": "46d32f5a9ffb420ea9868c2a71706c91", "name": "keystone"}, {"endpoints": [{"region_id": "RegionOne", "url": "http://10.0.100.45:8080", "region": "RegionOne", "interface": "admin", "id": "1532e78bfdde47eba67fd9463e40e669"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "internal", "id": "65d73a94d0cd40c4bf141743fa71505d"}, {"region_id": "RegionOne", "url": "http://10.0.100.45:8080/v1/AUTH_8416cab8b7b942b7bf76026fbe72cec9", "region": "RegionOne", "interface": "public", "id": "6516c825e37040e69d93b8928f105f19"}], "type": "object-store", "id": "742ad43e97db4ef5b00c442cc0f8b148", "name": "swift"}], "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "Administrator", "name": "Administrator"}, "audit_ids": ["BWgrdz-ZReeAtcjYOL1IUw"], "issued_at": "2015-11-16T12:13:37.083201Z"}}
				

"These are my personal views and do not necessarily reflect that of my employer"