Help! My Instance is Down!

Running into an error screen or unable to access the web app? Run through the following checks first.

This guide contains:

1. God Process

The first thing we're going to check is whether the god process monitoring service is running.

Run the following command in the console:

$ god status

This command should return:

cic-unicorn: up
dradispro-unicorn: up
nginx: up
resque:
cic-resque-1: up
dradispro-resque-1: up

If the command fails, try restarting with:

$ sudo /etc/init.d/god restart
$ god status

Command still not returning correctly? Check out the relevant log files by running:

$ vi /var/log/god.log

2. Web Applications

God process are working? Let's check whether the Web applications are running.

Unicorn is the application server so we're going check how many "unicorn master" processes are running with the following commands in the console:

$ ps aux | grep "unicorn master"

The process starting with 'root' corresponds with the CIC (the Combat Information Center, also known as the Dradis Administrator Console) and the process starting with '1000' corresponds to Dradis. The expected output looks like:

1000  19427  0.0  2.2 217780 92556 ? Sl Oct02 0:17 unicorn master
--env production --config-file config/unicorn.rb --daemonize config.ru
root 22017 0.0 0.7 105016 29876 ? Sl Jul21 0:02 unicorn master
--env production --config-file config/unicorn.rb --daemonize config.ru

Troubleshooting the CIC:

Did the "unicorn master" check uncover an issue with the CIC process? Time to dig into the relevant log files!

  • /var/log/god/cic-unicorn.log
  • /opt/dradispro/cic/shared/log/unicorn.err.log
  • /opt/dradispro/cic/shared/log/unicorn.out.log

To access from the command line, just run this command with any of the filepaths above:

$ vi FILEPATH

Troubleshooting Dradis:

Did the "unicorn master" check uncover an issue with the Dradis process? Time to dive into the log files!

  • /var/log/god/dradispro-unicorn.log
  • /opt/dradispro/dradispro/shared/log/unicorn.err.log
  • /opt/dradispro/dradispro/shared/log/unicorn.out.log
  • /opt/dradispro/dradispro/shared/log/production.log

To access the log files from the command line, just run the following command with any of the complete filepaths above:

$ vi FILEPATH

Specific Errors:

This section includes specific errors that you might encounter and the possible solution for resolving the error.

404 Not Found - nginx

Most likely this issue is caused by a down unicorn service and the god process check is giving you a false positive. Try starting up the dradispro-unicorn server using the command:

$ god start dradispro-unicorn

If nothing has changed, check the unicorn error logs. It should tell you why the server is failing to start:

$ vi /opt/dradispro/dradispro/shared/log/unicorn.err.log

Missing/Broken assets

If you get an error about a missing web file (js/css/html) or image file, it might be that something went wrong with your instance that an asset is now missing. To fix this, try running the following commands to compile the assets and restart the server:

$ RAILS_ENV=production bundle exec rails assets:precompile
$ god restart dradispro-unicorn

Site cannot be reached!

If the site is unresponsive and unreachable, it might be that the nginx server is down. Run the following command to check if the nginx service is running:

$ ps ax | grep nginx

If the service is not there, try starting up the nginx service using the command:

$ sudo service nginx start

If that didn't work, there might be an issue with your custom nginx configuration. The relevant files can be found in /etc/nginx/sites-*. The following command should help test the configuration:

$ sudo service nginx configtest

Ruby error

You might get an Internal Server Error (500) and an error message along the lines of NameError: undefined local variable or NoMethodError: undefined method.

Unfortunately, this might be a bug in the code and the only way to resolve this is patch to the server. If you encounter this, note down the steps you took to reach this error and contact the Support team so we can try to replicate this and resolve this issue right away.


3. Connectivity

No smoking guns with the processes or in the log files? Let's check your connectivity to make sure the VM can reach our activation server.

Ping it

Our activation server is at https://portal.securityroots.com. To test the connectivity, run the following from the console as dradispro:

ping portal.securityroots.com

If the command succeeds (transmits and receives packets), you've got connectivity! Continue on to the next step.

If the command fails, it looks like we need to establish connectivity first.

Check the DNS configuration in `/etc/resolv.conf` and update the DNS servers. You can try adding this content:

nameserver 8.8.8.8
nameserver 8.8.4.4

Run the ping command again to confirm that you've established a connection with the activation server.

cURL it

Run the following command from the console as dradispro:

$ curl -i https://portal.securityroots.com

This command should return the remove page.

If it doesn't, it may be that DNS is working but traffic is being stopped (by a firewall?). If you know who to contact internally to get this solved, please let them know that you need access to tcp/443 on portal.securityroots.com.

Adapt it

If the ping command resolves to "host unknown", make sure that your Dradis image adapters are set up properly.

Your Dradis image should have two adapters, NAT and Host-only. They should already be set, but make sure to check them and make changes if needed. If they are set up correctly, go to step 3.

  1. To add a Host-only interface, open VirtualBox's Preferences > Network. You can use the default values. (remember to enable the DHCP Server)

  2. Once that is done, update the Dradis image's Network settings with the NAT adapter and another adapter for the Host-only network (choosing the one you made in Step 1).

  3. Next, you'd need to make sure the two interfaces (eth0 and eth1) are configured. You can do that by editing the file /etc/network/interfaces with your preferred text editor and make it look like this:

    # This file describes the network interfaces available on your system
    # and how to activate them. For more information, see interfaces(5).
    
    # The loopback network interface
    auto lo
    iface lo inet loopback
    
    # NAT
    auto eth0
    iface eth0 inet dhcp
    
    # Host only
    auto eth1
    iface eth1 inet dhcp
    
    # The primary network interface
    allow-hotplug eth0
    iface eth0 inet dhcp
  4. Check if the interfaces are up with the command

    $ ip a
  5. Check if any interfaces are DOWN and UP them with the command

    $ ip link set <interface_name> up

    Example response: $ ip link set eth1 up

  6. Restart the instance and test with the command

    $ ping portal.securityroots.com

4. Contact Us

Still running into issues after walking through the 3 steps above? Please reach out to our Support team and we'll give you a hand.

InfoSec project delivery 5-day crash course

Learn innovative, actionable techniques and approaches for reducing the overhead that drags down InfoSec project delivery. You’ll learn how to optimize:

  • Scoping
  • Scheduling
  • Project Planning
  • Delivery
  • Intra-team Collaboration
  • Reporting and much more...

Your email is kept private. We don't do the spam thing.