LDAP. Easy on Ubuntu:

• apt install libpam-ldapd
• cd /etc/pam.d and fix common-account, common-auth, common-password, common-session
• fix /etc/nsswitch.conf
• fix /etc/security/access.conf
• pam-auth-update
• service nscd restart

In fact, with the pam-auth-update we probably don't even have to alter the files in /etc/pam.d ourselves.

Update: yep, it is very simple with Ubuntu:

apt install libpam-ldapd
pam-auth-update
service nscd restart

It eventually turned out to be quite simple on SmartOS, after hours spent trawling through guides for Solaris 10 & 11 and various manpage entries.

We start in the same way as for Ubuntu: install the LDAP plugin for PAM:

pkgin in pam-ldap

Then run vim /etc/pam.conf and add the LDAP module in where appropriate:

# grep ldap /etc/pam.conf 
login   auth sufficient         pam_ldap.so.1
other   auth sufficient         pam_ldap.so.1
other   account sufficient      pam_ldap.so.1
other   password required       pam_ldap.so.1

and edit /etc/openldap/ldap.conf with the relevant details. I'm not sure how important this is as it is probably superseded by the following stage.

We initialise the LDAP client with the following command (I stuck it in a script for easy editing and retrying after ldapclient uninit):

ldapclient manual \
-a credentialLevel=proxy \
-a authenticationMethod=simple \
-a defaultSearchBase=dc=mydomain,dc=com \
-a domainName=mydomain.com \
-a defaultServerList=<IP of LDAP server> \
-a proxyDN=cn=admin,dc=mydomain,dc=com \
-a proxyPassword=secret \
-a attributeMap=group:gidnumber=gidNumber \
-a attributeMap=passwd:gidnumber=gidNumber \
-a attributeMap=passwd:uidnumber=uidNumber \
-a attributeMap=passwd:homedirectory=homeDirectory \
-a attributeMap=passwd:loginshell=loginShell \
-a attributeMap=shadow:userpassword=userPassword \
-a objectClassMap=group:posixGroup=posixgroup \
-a objectClassMap=passwd:posixAccount=posixaccount \
-a objectClassMap=shadow:shadowAccount=posixaccount \
-a serviceSearchDescriptor=passwd:ou=people,dc=mydomain,dc=com \
-a serviceSearchDescriptor=group:ou=groups,dc=mydomain,dc=com \
-a serviceSearchDescriptor=shadow:ou=people,dc=mydomain,dc=com

Hopefully by now you can run ldaplist and getent passwd john, where john is a valid LDAP uid. Easy, wasn't it? :P

This allows us to sudo su to the user, but does not allow the user to log in by themselves. Running a tcpdump on the connection shows that SmartOS is looking for some additional Solaris-specific objectClasses:

Filter: (&(objectClass=SolarisUserAttr)(uid=test))
Filter: and item: equalityMatch
> attributeDesc: objectClass
> assertionValue: SolarisUserAttr
Filter: and item: equalityMatch
> attributeDesc: uid
> assertionValue: test

So we need to add a SolarisUserAttr object class to our schema or remove the SolarisUserAttr requirement from the SmartOS container.

...or not:

Solution: point 10 of https://docs.oracle.com/cd/E23824_01/html/821-1455/setupproblems-3.html#setupproblems-9 suggests that it's a case of having the wrong password format. Seems that Solaris only likes crypt...

In fact, here we have the contents of /etc/security/crypt.conf:

1       crypt_bsdmd5.so.1
2a      crypt_bsdbf.so.1
2b      crypt_bsdbf.so.1
md5     crypt_sunmd5.so.1
5       crypt_sha256.so.1
6       crypt_sha512.so.1

and /etc/security/policy.conf tells us:

CRYPT_ALGORITHMS_ALLOW=1,2a,md5,5,6
...
CRYPT_DEFAULT=5

Having set fusiondirectory to store the hash as crypt_sha512, we are able to log in locally and remotely to both ubuntu and smartos.