Web browsers have supported custom plug-ins and extensions since the 1990s, giving users the ability to add their own features and tools for improving workflow or building closer integration with applications or databases running on back-end servers.

The Google Chrome browser supports extensions that add to its functionality and which are typically hosted on the Chrome Web Store, but it is often desirable for firms to develop and host their own extensions internally.

According to Google’s Alternative Extension Distribution Options, Chrome extensions that are developed and hosted on a firm’s internal website are known as external extensions. This is slightly confusing at first, but external refers to the extension being external to the Chrome Web Store, not being external to the company that developed it.

We wanted to host our own Chrome extensions on an internal web server for web browsers running on the Linux operating system. However, despite setting up an example extension and following the Linux hosting requirements precisely, we would receive the following error when attempting to install the extension in the browser:

Package is invalid: CRX_REQUIRED_PROOF_MISSING

The error was devoid of explanation or reason, leaving little to go on. Some research on the web revealed that many people had complained about this error but each example found seemed to be for different reasons that did not match our case. On the road to a solution we passed many landmarks, each time expecting either success or at least a different, more informative error message. Unfortunately, each step we took revealed no further information, no clue that we had even progressed an inch, like we were trying to guess the secret password to enter Aladdin’s cave. “CRX_REQUIRED_PROOF_MISSING” was the cryptic greeting every time.

We did, eventually, solve the conundrum. For the benefit of others attempting the same feat, this blog post will walk you through how to install Chrome extensions from an internal web server. The instructions will have a heavy leaning toward Linux, although some of the lessons learned will apply to other operating systems.

Build an example extension

Follow the Getting Started Tutorial to build an extension you can test with. When this extension is built, dragging and dropping it into the chrome://extensions page will install the extension.

The tutorial walks you through using Chrome’s Load unpacked button in order to install the extension directly from your development folder. While there is also a Pack extension button that will create a CRX file that contains your extension, you may wonder, as we did, how to create a CRX file from the command-line. The CRX file format changed from CRX2 to CRX3 during 2019, leaving many scripts that you can find while trawling the internet broken.

To pack an extension from the command line, you can use the browser’s --pack-extension option:

$ chrome --pack-extension=<extension directory>

which will generate a new private/public key pair saving a new .crx and .pem file in the current directory, or:

$ chrome --pack-extension=<extension directory> \
         --pack-extension-key=<extension pem file>

to use an existing key file. More details on packaging can be found here.

You will need to obtain the extension ID and make a note of it. This is the unique identifier that Chrome will use to refer to your extension and will be required in some configuration files later on. If you install the extension into Chrome by dragging and dropping, then Chrome will display the extension ID for you. Otherwise, to do this programmatically using the .pem file, see here.

Unfortunately, Chrome on Linux expects to have an X display for the --pack-extension command even though it does not open a window. Also the --headless option does not seem to work with --pack-extension. So if you are trying to get this to work on a server that has no X display, I have found that Xvfb makes it possible, e.g.

$ Xvfb &
$ DISPLAY=:0 chrome --pack-extension=<extension_directory>
$ kill %1

Setting up a test web server

Configure for SSL connections

Next you will need a web server with an SSL configuration. We used nginx which was quick to compile, install and configure. The web server needs to be configured to listen for SSL connections (usually on port 443).

Configure MIME types

Make sure that the mime.types file is correctly configured for the following file extensions:

text/xml                                         xml;
application/x-chrome-extension                   crx;

Create SSL certificates

To get Chrome to trust SSL connections to the test web server, create a small certificate chain: a server certificate signed by a test CA certificate that you load into the Chrome browser as a trusted certificate authority.

To create the CA certificate, start with a ca.conf file like this:

[req]
prompt = no
default_md = sha256
distinguished_name = dn

[dn]
C = <country_code>
ST = <state>
L = <locality>
O = <organisation name>
CN = My Test Root CA

We will use this configuration file in a moment. You will also need a server.conf file that looks like this:

[req]
prompt = no
default_md = sha256
x509_extensions = v3_req
distinguished_name = dn

[dn]
C = <country_code>
ST = <state>
L = <locality>
O = <organisation name>
CN = %HOSTNAME%

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = *.<domain>

This will be used to create an extended X.509 certificate with a subjectAltName attribute, required by Chrome browsers. The alt_names section may contain DNS.2 and DNS.3 and so on for as many domain names that your web server is going to be answering for. The %HOSTNAME% text can be left as-is, this will be substituted for the real hostname below and allows for the process to be easily scripted.

Now you have the ca.conf and server.conf files, you can use OpenSSL to generate the certificates you need. We will produce these files inside keys and certs subdirectories, so create these first and keep them secure:

$ mkdir -m 0700 keys certs

Now either run the individual commands provided below, or you may shortcut the process by running this generate-ssl-cert script.

Create a new CA public/private key pair and X.509 certificate:

$ root_key=keys/rootCA.key
$ root_crt=certs/rootCA.crt
$ root_days=730
$ openssl req -x509 -config ca.conf \
        -newkey rsa:4096 -nodes -keyout $root_key -out $root_crt \
        -days $root_days

You can view the new certificate with:

$ openssl x509 -in $root_crt -noout -text | less

Now use OpenSSL to generate a new server private/public key pair and a certificate signing request (CSR):

$ hostname=<fully-qualified hostname>
$ host_key="keys/$hostname.key"
$ host_csr="certs/$hostname.csr"
$ openssl req -new -reqexts v3_req \
        -config <(sed "s/%HOSTNAME%/$hostname/" server.conf) \
        -newkey rsa:4096 -nodes -keyout "$host_key" -out "$host_csr"

Finally, sign the CSR with the CA private key and generate the server certificate:

$ host_crt="certs/$hostname.crt"
$ host_days=365
$ openssl x509 -req -in "$host_csr" -extensions v3_req \
        -extfile <(sed "s/%HOSTNAME%/$hostname/" server.conf)  \
        -CA $root_crt -CAkey $root_key -CAcreateserial \
        -days $host_days -out "$host_crt"

You can view the new certificate with:

$ openssl x509 -in $host_crt -noout -text | less

Move the server key and certificate into the locations specified in the web server configuration, and start/restart the web server.

Import CA root certificate into Chrome browser

Now you need to add the self-signed CA root certificate (rootCA.crt) into your test Chrome web browser. Open chrome://settings/certificates, click on Authorities and then Import. Locate the CA certificate and when prompted for the trust settings, check all of the available boxes.

Confirm that you can view the web server’s index.html document over HTTPS. Chrome shouldn’t complain about the SSL certificate not being trusted, there should be a closed padlock symbol to the left of the URL in the address bar. If you click on the padlock symbol, it should say in green: Connection is secure.

Set up web server documents (CRX/XML)

You will need to place the CRX file (packed extension) you created earlier into the web server’s documents directory. You will also need to create an XML file that describes the location of the CRX file, like this, which you also place on the web server:

<?xml version='1.0' encoding='UTF-8'?>
<gupdate xmlns='http://www.google.com/update2/response' protocol='2.0'>
  <app appid='<extension ID>'>
    <updatecheck codebase='https://<fully-qualified hostname>/<filename>.crx' version='1.1' />
  </app>
</gupdate>

At the time of writing, the Linux hosting page was erroneously quoting that the gupdate tag in this XML document should refer to an https URL. This is not true. The gupdate tag must use the http URL as above. This URL is not actually followed by the browser but is only used as a hint to the parser about the XML structure, as seen here in the Chromium source code.

Now you need to edit the manifest.json file inside your Chrome extension and add the following key which points to your XML file:

"update_url": "https://<fully-qualified hostname>/<filename>.xml",

Re-pack your extension with the updated manifest to the .crx file, remembering to use the .pem file from earlier so that the extension ID remains the same, and copy into place on the web server. If you forget to use the .pem file then a new public/private key pair is generated and as the extension ID is computed from the public key the ID would change as a result, which is generally not what you want.

Chrome enterprise policies

To allow your extension to be installed manually, or to have it forcibly installed, you will need to set the appropriate policies.

ExtensionInstallSources must be configured with URLs or wildcards matching the web address where the extension is hosted as well as the web address that contains the link to the extension if a user is expected to click on a link to install it (the referrer), e.g.

  "ExtensionInstallSources": [
    "https://<fully-qualified hostname>/*",
    ...
  ],

This caught me out for a while as the documentation made no mention of it, but you will not be able to install an extension by typing in, or copying and pasting, the URL of the .crx file into the browser’s address bar. Without the referrer URL in this policy you won’t be able to install the extension by clicking on a link.

You may wish to put a * in your ExtensionInstallBlacklist for ordinary users which disables the Load unpacked button in chrome://extensions. If ExtensionInstallBlacklist contains a * or any wildcard that would end up blacklisting the URL of your internal extension, then you must explicitly permit your extension ID in the ExtensionInstallWhitelist, e.g.

  "ExtensionInstallWhitelist": [
    "<extension ID>",
    ...
  ],

To forcibly install your extension you may add it to the ExtensionInstallForcelist policy. This policy line must point to the .xml file (not the .crx file), e.g.

  "ExtensionInstallForcelist": [
    "<extension ID>;https://<fully-qualified hostname>/<filename>.xml",
    ...
  ],

To confirm that the web browser has the expected policy configuration, you can view the current policy settings at chrome://policy.

If you are using the ExtensionInstallForcelist policy to install your extension, note that the moment you remove your extension ID from that policy it should be automatically removed from the browser.

Chrome policies per user on Linux

If you need to vary the Chrome web browser policy files by user on Linux, you’ll quickly discover that Chrome does not support this. However, it is possible to achieve this using /etc/namespace.conf, otherwise known as polyinstantiated directories.

Let’s say your policy file is called /etc/opt/chrome/policies/managed/my_policy.json. With polyinstantiated directories, it is possible to provide a particular tailored version of that file by user, as the PAM session module can overlay the directory according to a set of rules.

To do this, first create a directory where the source files live. For testing purposes, I put this under /etc/opt/chrome/policies/users. Run these commands as the root user:

$ cd /etc/opt/chrome/policies
$ mkdir -m 0 users
$ mkdir users/my_user
$ cp managed/my_policy.json users/my_user

The permissions on the parent directory have to be 000, as required by pam_namespace(8).

Now edit /etc/opt/chrome/policies/users/my_user/my_policy.json to contain the specific changes required for the user.

Lastly, configure pam_namespace to map this directory over the top of the original directory when that specific user logs in. This is done by appending the following line to /etc/security/namespace.conf. Warning! Before you do this make sure you have a terminal window open as root on your test host so you don’t accidentally lock yourself out if anything goes wrong!

/etc/opt/chrome/policies/managed    /etc/opt/chrome/policies/users/ user    ~my_user

The fields are delimited by whitespace. The first field is the target directory that will be replaced. The second field locates where the user-specific directories originate from. The third field specifies that the username should be appended to the second field to find the source directory. The fourth field starts with ~ and is a comma-separated list of all users this rule applies to. Alternatively, without the ~ prefix, this can be a comma-separated list of all users the rule does not apply to.

Now when I open another terminal window and login, as pam_namespace is already configured in the PAM stack, I see that /etc/opt/chrome/policies/managed/my_policy.json contains my user-specific modification.

If this is not working as expected, check that all of the appropriate files in /etc/pam.d are configured to require pam_namespace.so like this:

session    required     pam_namespace.so

Also watch out for incorrect syntax in /etc/security/namespace.conf. The directory in the first field must exist already and the second field must end with a slash. If anything is wrong, the user won’t be able to login at all! In this event, you’ll not see much in /var/log/messages:

2020-04-17T18:09:54.297987+00:00 my_host systemd[1]: Started Session 679 of user my_user.
2020-04-17T18:09:54.298292+00:00 my_host systemd-logind[2345]: New session 679 of user my_user.
2020-04-17T18:09:54.400902+00:00 my_host systemd-logind[2345]: Removed session 679.

… but you should find something useful in /var/log/secure, for example:

# grep pam_namespace /var/log/secure
2020-04-17T17:55:01.344288+00:00 my_host sshd[24295]: pam_namespace(sshd:session): Mode of inst parent /etc/opt/chrome/policies not 000 or owner not root
2020-04-17T17:56:13.826555+00:00 my_host sshd[24583]: pam_namespace(sshd:session): Mode of inst parent /etc/opt/chrome/policies not 000 or owner not root
2020-04-17T17:56:20.114737+00:00 my_host sshd[24608]: pam_namespace(sshd:session): Mode of inst parent /etc/opt/chrome/policies not 000 or owner not root
2020-04-17T17:57:38.896329+00:00 my_host sshd[24867]: pam_namespace(sshd:session): Error stating /etc/opt/chrome/policies/managed/test: No such file or directory
2020-04-17T18:08:39.280280+00:00 my_host sshd[27049]: pam_namespace(sshd:session): Error stating /etc/opt/chrome/policies/managed/test: No such file or directory

If you’re really stuck, you can add the debug argument after pam_namespace.so in the appropriate /etc/pam.d configuration file, which adds more verbose logging to /var/log/secure.

Summary

In summary, the main points to focus on in order to support installing Chrome extensions on Linux from an internal web server instead of the Chrome Web Store are:

  • Google make it intentionally difficult to host Chrome extensions on an internal web server, I presume for security reasons.
  • There is about one error you’ll ever get from Chrome when trying to install an extension from an internal web server and something isn’t configured right: CRX_REQUIRED_PROOF_MISSING.
  • Set-up a web server such as nginx to run an instance on port 443 for testing using a test SSL certificate signed with a self-signed CA cert that you import into Chrome as a trusted certificate.
  • The manifest.json file inside the Chrome extension must have an update_url parameter pointing to an XML file hosted on the internal web server.
  • The packed extension format changed from CRX2 to CRX3 in 2019 so many tools found on the web no longer work. Use chrome --pack-extension to do this on the command-line.
  • The XML file contains the extension ID, which is derived from the public key that accompanies the CRX file. Contrary to currently available documentation, the gupdate tag in the XML file must have the exact URL http://www.google.com/update2/response and not use https.
  • The web server must use the correct MIME type for CRX files: application/x-chrome-extension.
  • If you need to vary the Chrome policy file for different users, you must use polyinstantiated directories to achieve this as Chrome does not offer OS user level policies on Linux.
  • You cannot type in or copy/paste the URL of a CRX file into the browser’s address bar, you must instead click a link provided on a web page and that website must be permitted in the ExtensionInstallSources policy.