Troubleshooting NoMAD

While we make every attempt to ensure a trouble-free experience with NoMAD, there are times when your environment isn’t what NoMAD is expecting, or when NoMAD has a bug.

Here are a few tips and tricks to get things back on the straight and narrow when you do encounter these scenarios.

Verbose Logging

The first step in any troubleshooting process is to run NoMAD from the command line and put it into verbose mode. It’s perfectly acceptable to do this while other copies of NoMAD are running; you’ll just have multiple icons in the menu bar.

To launch NoMAD in verbose, open up a Terminal window and put the full path to the NoMAD binary, not just the application bundle with -v. For example, if NoMAD is installed in the Applications folder you’d use this in Terminal:

/Applications/NoMAD.app/Contents/MacOS/NoMAD -v

This will launch NoMAD and put all logs directly into the terminal window that’s already open. This makes it very simple to gather the NoMAD logs and see events happen in real time.

Most issues are at least identifiable in the logs. Some things may jump out, as verbose logging should record a lot of events and information as NoMAD works. The other issue that’s typically encountered is an “illegal instruction 4” error. This typically represents a variable in the code that was expected to have a value, but is empty. NoMAD has a number of checks for these, but there’s always the chance that either we’ve left something out, or you have a setting that we’ve never seen missing a value before.

Additional CLI Options

There are some other flags you can use when launching from the CLI as well:

  • “-prefs” – On version 1.1.4 and newer, this will print out all preferences as NoMAD sees them, including whether or not they are being forced via a configuration profile.
  • “-rawLDAP” – This will list all LDAP queries and responses as NoMAD works. You’ll get to see the actual ldapsearch syntax being used and the full LDIF response from the LDAP server.
  • “-shares” – This will list all file shares as seen by NoMAD and give you more verbose output specific to the file shares menu.

All of these options can be used together or individually when launching NoMAD from the CLI.

Configurations

The vast majority of issues we see with organizations using NoMAD are due to configuration issues. Most of these configuration issues are in turn caused by creating malformed configuration profiles, scripting defaults instead of using a configuration profile, or managing preference keys that aren’t meant to be managed.

The simplest way to test for this is to pull the configuration and see if the problem persists. If the problem is with the configuration itself, the next easiest step is to attempt to validate the XML, if you’re using a configuration profile.

A good general rule of thumb when dealing with configurations is to manage only as much as you absolutely need to. As an example, you generally shouldn’t have any settings in NoMAD that are set to a blank value.

Network and AD

The next most common problem is with either the network or the AD configuration that’s in use. By now, NoMAD is rather resilient to any network outages and other issues. More common are more “exotic” AD configurations. While we’ve seen a lot of these and have work arounds for most, there are always new ones.

More Troubleshooting Tips

So, you’ve kinda-sorta got it working, but it’s not doing what you want… or it’s just flat out failing. Here’s how to replicate what’s going on manually to find out where things have gone awry.

1. SRV Records

The first thing NoMAD does is attempt to look up your AD Domain Controllers via a DNS lookup for any LDAP SRV records. For the following examples, we’ll be using “company.com” as your AD domain and “server.company.com” as your AD Domain Controller. If you’re playing along at home, swap your real information in for these examples:

dig +short -t SRV _ldap._tcp.company.com

This will query DNS for any LDAP SRV records. If you don’t get any results, this means that you’re a) not able to reach your corporate AD domain, at which point it’s probably a VPN issue or general network issue, or that b) you can’t get any DNS resolution for your domain. Make sure that you’re using the expected DNS servers, or that you can actually reach your internal network.

2. Kerberos Login

Once you’ve gotten some results of the SRV lookup, the next step is to attempt to get a Kerberos ticket with an AD username and password:

kinit user@COMPANY.COM

No news is good news here, but you can test to ensure that it worked by using the klist command, which should show that you have a Kerberos TGT for your domain.

If this doesn’t work, it’s most likely that you are once again unable to reach any of the AD Domain Controllers.

3. LDAP Queries

If you were able to login via Kerberos, you can try looking up information via LDAP.

Use any of the servers that you find via the dig command in the first step and attempt to do an LDAP query against it:

ldapsearch -LLL -Q -H ldap://server.company.com -s base defaultNamingContext

This should return a chunk of text with a defaultNamingContext attribute in it. This lets you know that you can in fact use that username and password to lookup information via LDAP.

If this doesn’t work, you’re probably either not on the internal network, or you may have some funky access rights in AD that will cause further issues.

4. Windows CA Troubleshooting

Listed below are few top tips for working with Windows Certificate Authorities (CA). Much like the rest of NoMAD there isn’t any real magic to this as well. NoMAD will leverage your AD credentials to request a certificate from the CA’s web portal. In order to regress any issues you can manually attempt this process by just using Safari.

  • Use NoMAD to ensure you are logged in with your AD user and that you have a valid Kerberos TGT
  • Use Safari, as it's Kerberos-aware by default, to connect to the CA web portal. This typically be in the style of ```https://x509.domain.com/certsrv```
  • You should be automatically logged in. If you are presented with a security dialog about trusting the CA's SSL cert, this may be where things are going wrong. Import that CA's root cert into your local keychain and trust it. NoMAD won't connect if the web server is untrusted. If you're not automatically logged in, the web portal is most likely not set to use "Windows Authentication" which is just a checkbox in the Microsoft IIS settings. Without this set, the web portal will be asking for a password and not a Kerberos ticket.
  • If all of that works, you should be able to request a certificate through the web portal. Note that some earlier versions, i.e. pre-Windows 2008, of the portal leveraged ActiveX in the web page and may not work on a Mac. When you go through this process, you should be able to specify a certificate template to use. You should use that same template name in NoMAD.

Reporting Issues

If you’re a support customer, please use the support email and we’ll get to work on figuring out what’s going on for you. In addition, or if you don’t have a support contract, you can file an issue on the NoMAD GitLab page or hop into the #nomad Slack channel at macadmins.slack.com.