CQ 5.5: Rien ne va plus - I need a text Shell

Issue

You inadvertently stopped the HTTP Service or the Web Console and cannot access the system any longer. Or, your repository crashed and the Web Console just does not let you easily fix the repository. Or, you are looking for the CRX Console from earlier versions of CRX.

Resolution

This document explains how you can install a text based command line shell helping you recover from these and maybe other problems.

Key Players

The key to the solution is to use the command line interface to the OSGi framework which is provided by the Apache Felix Gogo Shell. As is custom in the OSGi world, the Gogo Shell comes along as a collection of bundles:

Apache Felix Gogo Runtime -- The core shell runtime. This bundle provides the core API and integration functionality to run a command shell.
Apache Felix Gogo Command -- The basic commands for OSGi framework maintenance. This bundle provides commands to introspect and manipulate and OSGi framework and its bundles.
Apache Felix Gogo Shell -- The text shell functionality. This bundle provides a basic, local shell (disabled by default in Granite based applications) on which the remoting features are based.
Apache Felix Web Console Gogo Shell Plugin -- Text Shell support inside the Web Console. This bundle extends the Gogo Shell bundle by exposing the text shell in the Web Console.
Apache Felix Remote Shell -- Telnet-style Text shell support. This bundle extends the Gogo Shell bundle by exposing the text shell over TCP/IP.

More information on the Gogo Shell is available from the Apache Felix site.

Required Basics

The core bundles -- Runtime, Command, and Shell -- are essential and are always required.

Web Console Access

If access to the Web Console is still possible, using the Web Console Gogo Shell Plugin is recommended for remote shell access.

Telnet Access

If access to the Web Console is not possible any longer -- e.g. the Http Service is dead or the Web Console itself does not work any more -- you may install the Remote Shell bundle.

Unlike the Web Console Gogo Shell Plugin, the Remote Shell bundle does *not* allow for true remote access. For security reasons, the Remote Shell bundle by default listens to port 6666 on localhost (127.0.0.1) only. That, access is only possible from the same box.

Security Considerations

Web Console

Access to the Web Console Gogo Shell Plugin is guarded as everything in the Web Console, through simple authentication and basic access control. For example the Sling Web Console Security provider out of the box only allows for the admin user to access the Web Console.

As such, the Web Console should, of course, also not be exposed through the Dispatcher to the public internet.

Telnet

The Remote Shell bundle does not implement any access control whatsoever. The only basic security implemented is, that by default only access to localhost (127.0.0.1) is possible. This means, A remote shell to the host (e.g. an SSH session) is required before being able to telnet into the Remote Shell.

Also, the Remote Shell bundle is not intended to be active by default and over time. The bundle should be installed in case of emergency and should be uninstalled after solving the emergency problem to prevent potential issues. No regular functionality should be based on the Remote Shell bundle alone.

Installation

Content Package Installation

If the JCR repository is still operable and is accessible over HTTP, the easiest thing to do is to install the attached Gogo Shell Content Package. This content packages provides the Gogo Runtime, Command, and Shell bundles as well as the Web Console Gogo Shell Plugin.

It does not include the Remote Shell because of security concerns and because it is not really needed in this situation.

Web Console Installation

If the JCR Repository is not available any more but the Web Console is accessible, you can download and unpack the attached Gogo Shell Content Package and upload the contained bundles through the Web Console.

Again, the Remote Shell bundle should not be installed in this case because of security concerns and because it is not really needed.

Emergency Installation

In case of solving emergency problems where neither the JCR Repository nor the Web Console are operable and accessible, you will have to fall back to local installation of the Gogo Shell. This installation requires access to the filesystem and shell access.

The basic installation steps are as follows:

Get a text based session to the system running the Granite application
Create an install folder in the Granite application folder (generally below crx-quickstart
Download and unpack the attached Gogo Shell Content Package
Copy the Gogo Runtime, Command, and Shell bundles into the crx-quickstart/install folder
Download the attached Remote Shell bundle and copy it into the crx-quickstart/install folder

The OSGi Installer running in the Granite based application will automatically pick up the bundles from the crx-quickstart/install folder and install and start them. You might have to wait a few seconds for this to happen.

After the installation, you can telnet in to the Gogo Shell which listens on port 6666 by default:

$ telnet localhost 6666
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
_
Welcome to Apache Felix Gogo

g!

If all goes well, you get the command prompt -- g! -- from the Gogo Shell where you can start running commands like help (to get a list of commands available) and lb (to list running bundles).

See the the Apache Felix site for more information on the Gogo Shell.

Recovering from a stopped Http Service Bundle

To recover from inadvertedly stopping the Http Service bundle follow these steps:

Install the Gogo Shell bundles as explained above in the section Emergency Installation above
Start the shell session using the telnet command:
$ telnet localhost 6666
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
_
Welcome to Apache Felix Gogo

g!
Use the lb command to list the Http Service bundle. We assume this bundle has http service in its name, so we use this (quoted) as the argument to the lb command :
g! lb "http service"
START LEVEL 30
ID|State |Level|Name
21|Resolved | 5|Day CQSE HTTP Service (4.1.32)
g!
From the output we see that in fact the CQSE HTTP Service bundle is not active. So we use the start command using the bundle's ID of 21 as the argument:
g! start 21
g!
So far so good. Lets check the outcome:
g! lb "http service"
START LEVEL 30
ID|State |Level|Name
21|Active | 5|Day CQSE HTTP Service (4.1.32)
g!
Use the lb command to check other bundles are also started. Particulary you will want to verify the Web Management console and plugins are active:
g! lb console
START LEVEL 30
   ID|State      |Level|Name
   24|Active     |    5|Apache Felix Web Management Console (3.4.1.R1334005)
   25|Active     |    5|Apache Felix Web Console Service Component Runtime/Declarative Services Plugin (1.0.0.R1201749)
   26|Active     |    5|Apache Felix Web Console Event Plugin (1.0.2)
   27|Active     |    5|Apache Felix Web Console Memory Usage Plugin (1.0.2)
   28|Active     |    5|Apache Felix Web Console Package Admin Service Plugin (0.0.1.R1233342)
   55|Active     |   15|Apache Sling Web Console Security Provider (1.0.0)
   74|Active     |   20|Adobe Granite Workflow Console (0.0.4)
138|Active     |   20|Apache Felix Web Gogo Shell Plugin (0.0.1.R1238480)
g!
Everything looks fine here. So nothing more to do.
You may now exit the shell by typing Ctrl-D
You should now uninstall the Remote Shell bundle by removing it from the crx-quickstart/install folder (you can still reinstall it again should it be required again).

You should now be able to access the application with your browser.

Access the Repository

Starting with CRX 2.3.18 (included in CQ 5.5 Update 1) the repository can be accessed using the gogo shell. This can be done by using the telnet command described above or using a webbrowser (http://localhost:4502/system/console/gogo). All CRX related commands have a crx: prefix. You can list these by typing crx:help

Available commands:
add         Adds a new node or child node entry.
bundlecheck Checks node bundles
cd          Changes the current work node.
check       Checks the repository
checkin     Does a checkin on a node.
checkout    Does a checkout on a node.
clone       Clones a node.
connect     Connects to a CRXRepository.
cp          Copies a node.
echo        Prints the arguments to the console
help        Prints this help
login       Perform a login on the repository.
logout      Perform a logout on the repository.
ls          Prints the list of nodes of properties.
mixin       Manipulates the set of mixin nodetypes of a node.
mv          Moves a node.
patch       Patch lowlevel data of an item state.
print       Prints the definition of a node or a property.
pwd         Prints the current path.
refresh     Refreshs a node, property or session
rm          Removes a node or property.
save        Saves a node, property or session
spool       Spools (copies) a workspace.
stat        Prints the lowlevel (persistence) information of a node or a property.
workspace   Manages workspaces

Paths:
Paths can be specified either
- relative to the cwd (i.e. './foo/bar')
- absolute (i.e. '/foo/bar')
- or uuid-relative (i.e. '[cafebabe-cafe-babe-cafe-babecafebabe]/foo/bar')

Use 'help -d' to list the details for all commands or
use 'help <cmd>' for a specific command.

If you want to fix repository inconsistencies you can try:

Start the console
$ telnet localhost 6666
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
_
Welcome to Apache Felix Gogo
login to the default workspace using admin credentials
g! crx:login admin admin
successfully logged in as admin in workspace crx.default
display the description of the bundlecheck method
g! crx:help bundlecheck
Synopsis:
bundlecheck [options] [path | uuid]

Description:
Checks either the current node's bundle, the bundle for a given node path, the bundle
for a node UUID or all stored bundles (regardless of hierarchy information). This command
only works with bundle persistence managers (eg. DB bundle and TarPM).

Options:
-a check all bundles in the PM storage (-r does not apply)
-r recursively check subnodes
-f try to fix missing child node errors
-p print node paths and verify parent relations (slower)
-x check node references (slower)
-X check node references and remove invalid (slower)
-e only print errors (faster)
-o <file> dump output to file.
-l <path> move orphaned nodes to this lost+found parent node
display the root directory
g! crx:ls
+ /
+ bin
+ rep:repoPolicy
+ rep:policy
+ jcr:system
+ var
+ libs
+ etc
+ apps
+ home
+ tmp
+ content
- sling:resourceType
- jcr:mixinTypes
- sling:target
- jcr:primaryType
create a lost+found folder
g! crx:add lost+found
save the session
g! crx:save
recheck the root directory structure (verify the lost+found folder)
g! crx:ls

+ /
+ bin
+ rep:repoPolicy
+ rep:policy
+ jcr:system
+ var
+ libs
+ etc
+ apps
+ home
+ tmp
+ content
+ lost+found
- sling:resourceType
- jcr:mixinTypes
- sling:target
- jcr:primaryType
perform the bundlecheck:
g! crx:bundlecheck -rfel lost+found
Using node b006af08-d78c-44b3-b847-58472e7962cc as parent for orphaned nodes.
checked 1000 nodes with 0 error(s) so far.
...
checked 66000 nodes with 0 error(s) so far.
Consistency check of 66717 nodes finished in 110015 milliseconds.
No errors found.