Installing a simple dCache setup
This article illustrates how to install a basic dCache system on one single (virtual) machine. The procedure should be adequate for all versions of dCache, but of course dCache.org may introduce modifications or new configuration issues any time. So this procedure has been valid (that means, it has been tested) for dCache versions 1.9.3 and 4 (including all subversions). The respective versions for Java and PostgreSQL have been 1.6.0_13 and 8.3.7.
For this lecture we will utilize the free virtualization software VirtualBox and the project dCacheToGo. Hence please have a look on this VirtualBox wiki page for instructions on how to set up and use a virtual machine with VirtualBox.
dCache is completely written in Java. In order to run dCache, all machines must have installed the Java Development Kit in version 1.5 or higher.
Using a non-Sun-JRE should be possible, but it has not been widely tested.
For few services dCache uses databases. PNFS/Chimera and SRM are the most important services that use databases, for some others databases are only optional. dCache.org suggests to use the open source software PostgreSQL – here version 8.3 or higher is recommended.
At this point this manual assumes you have a machine that meets all prerequisites mentioned before. As participant of the dCache administration workshop you will be provided with a ready-made image to build your virtual machine.
From now on we will edit flat files several times. Therefore we advise to use the editor vi. Start it by stating
# vi <file to edit>
on the command line. If a file is not existing, it will be created, otherwise opened. To toggle between vis read and edit mode press 'i' or 'ESC' respectively. You can find a variable quickly by typing '/' directly followed by the string to search for (e.g. "/SpaceManager" will match srmSpaceManagerEnabled) - the capitalization matters!
PostgreSQL must be installed on every server, that hosts a dCache service with database backend (although there are possibilities to set up the database apart of the dCache domain, that uses it). At least the following rpms of PostgreSQL are needed:
After one has installed these rpms (participants of the dCache administration workshop received an ready-made image with preinstalled PostgreSQL), the database has to be initialized, configured and started, before dCache can be installed.
- Initialize postgres with
# service postgresql initdb
- Open /var/lib/pgsql/data/pg_hba.conf and change following lines near the end of the file
# "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: host all all 127.0.0.1/32 trust
- Start the postgresql server
# service postgresql start
- One might find it usefull to let postgres start automatically on reboots (which is needed definitely, if you want to make dCache start on boot-up, too).
# chkconfig postgresql on
According to the purpose each node has to fulfill, not every dCache component has to be installed on each node. The installation packages can be found at dCache.org. However, during this dCache workshop we will install the whole dCache software on a sole (virtual) machine. If you are working on the offered ready-made image you can find all required rpm-packages in the directory /root/install.
# rpm -ihv /root/install/dcache*
This package includes all the dCache components that dCache consists of.
dCache Clients (dcap, SRM)
The client package is needed on all nodes that are meant to offer access to dCache setups.
dCache.org provides yum repositories with all the RPM packages from above. The configuration file (for example /etc/yum.repos.d/dcache.repo) to add these repositories is:
[dcache.stable] name=dcache.stable baseurl=http://cvs.dcache.org/repository/yum/sl4.4.0/$basearch/RPMS.stable/ enabled=1
node_config and dCacheSetup
After the dCache rpms are installed, the setup has to be configured properly. For this mainly two central configuration files have to be edited: node_config and dCacheSetup. As starting point both are shipped as templates with dCache:
# cp /opt/d-cache/etc/node_config.template /opt/d-cache/etc/node_config # cp /opt/d-cache/etc/dCacheSetup.template /opt/d-cache/config/dCacheSetup
- This flat file defines few, node-specific variables that are needed on all nodes the dCache setup consists of. The following values have to be set in the node_config file:
|Defines the type of namespace dCache will use. Possible values are pnfs or chimera.|
NAMESPACE_NODE=<FQDN of your headnode>
|Defines the (fully qualified domain) name of the node that hosts the dCache domain for managing the namespace. Use the output of "hostname -f"; for all users of our VirtualBox image, this is equal to "dcachetogo.dcache.org".|
|Defines the role this node has to fulfill in the global dCache setup. Set this either to admin, so a predefined set of services will be launched on this host, or custom and select every single service by means of the next variable.|
SERVICES="dcap gsidcap gridftp info"
|List all services that this node should host. Possible values can be chosen from dcap, gsidcap, srm, gridftp, xrootd, info, gPlazma, chimera, admin, httpd, utility, dir, statistics, lm, dCache, replica.|
- In the dCacheSetup file almost all variables are of global importance. Hence, it is strictly advised to keep this file as much identical as possible among all nodes of the dCache setup (it will ease the administration a lot!). Set the following values inside of it:
serviceLocatorHost=<FQDN of your headnode>
|Adjust this to point to one unique server for one and only one dCache instance (usually the admin node).|
|Set this to the jdk executable you want to use for dCache.|
java_options="-server -Xmx64m -XX:MaxDirectMemorySize=64m (...)"
|The numbers for Xmx and MaxDirectMemorySize indicate the amount of memory allocated for each java task initialized by dCache. As the machine we are working with may not have that much memory, we should decrease it beforehand. (during this workshop the dCache setup will include more than ten java tasks)|
|dCache needs to know about locations for all stored files. This variable decides whether this information is kept in a seperate database or the namespace service itself.|
|In principle dCache can write its log files in any existing directory. But as every dCache service has its own log file, thus the number may become high, it is beneficial to seperate them from the default system log files.|
The following variable does not exist yet, so add the whole line anywhere inside dCacheSetup
Users postgres and srmdcache
After postgres is installed, the databases are owned by the newly added user <tt">postgres</tt>. But dCache will connect to the databases as yet another user which is called srmdcache (by default). As srmdcache needs other rights for handling PostgreSQL databases like a normal unix user, we have to create him as user postgres with PostgreSQL tools (the option -U allows to execute commands with a different user account).
# createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt srmdcache <type in "srmdcache" as password two times> # createdb -U srmdcache dcache
Still as user postgres we need to create a database for Chimera, the namespace type of our choice in this workshop:
# createdb -U postgres chimera # psql -U postgres chimera < /opt/d-cache/libexec/chimera/sql/create.sql # createlang -U postgres plpgsql chimera # psql -U postgres chimera < /opt/d-cache/libexec/chimera/sql/pgsql-procedures.sql
The Chimera NFS server uses the /etc/exports file to manage exports. The typical export file looks like:
/ localhost(rw) /pnfs *.dcache.org(rw)
After you have edited /etc/exports, start the NFS server via script:
# /opt/d-cache/libexec/chimera/chimera-nfs-run.sh start
To automate the launching of that script at startup time, link it into /etc/init.d/ and announce it to chkconfig:
# ln -s /opt/d-cache/libexec/chimera/chimera-nfs-run.sh /etc/init.d/chimera-nfs-run # chkconfig --add chimera-nfs-run # chkconfig chimera-nfs-run on
To configure the server, we need to mount the administrative view of Chimera (localhost:/) locally (remote mounts are not allowed yet). This will allow us to create the main Chimera path.
# mount localhost:/ /mnt
First we create the root of the Chimera namespace, called pnfs (for legacy reasons).
# mkdir -p /mnt/pnfs/dcache.org
Now we need to add directory tags (metadata, which will be evaluated by dCache and inherited by future subdirectories):
# mkdir -p /mnt/pnfs/dcache.org/data # echo "chimera" > /mnt/pnfs/dcache.org/data/'.(tag)(sGroup)' # echo "StoreName chimera" > /mnt/pnfs/dcache.org/data/'.(tag)(OSMTemplate)'
The configuration is done now, so unmount NFSv3:
# umount /mnt
- Whenever you need to change the configuration, you have to remount the admin-view (localhost:/) to a temporary location like /mnt.
For produtcion we mount Chimera under /pnfs
# mkdir /pnfs # mount localhost:/pnfs /pnfs
Remember that you need to mount the name space at least on the SRM-node and the GFTP-node(s). Pools do not need a mount anymore.
According to our settings in the dCacheSetup file, we need to create the directory dCache is ordered to write the logfiles to.
# mkdir /var/log/dCache
So gsidcap can be used normally, the environment variable LD_LIBRARY_PATH has to be set. We create a wrapper around the regular dccp executable in order to achieve this.
# vi /usr/local/bin/dccp
#!/bin/sh LD_LIBRARY_PATH=/opt/d-cache/dcap/lib /opt/d-cache/dcap/bin/dccp "$@"
# chmod +x /usr/local/bin/dccp
Once all this configuration is done, we have to make it take effect:
- Everytime basic settings - e.g. there has been an update of the dCache version - have changed, install.sh must be executed!
dCache pools hold all the data ever written into dCache. These pools are completely independent from the PNFS directories (for the time beeing). In fact, they could be created anywhere and then mounted locally. In every case we want to seperate the data storage pools from the PNFS file system and create a new directory for them.
# mkdir /pools
The (logical) dCache pools are created with the /opt/d-cache/bin/dcache-tool (usage: /opt/d-cache/bin/dcache pool create <size> <target location>). E.g.
# /opt/d-cache/bin/dcache pool create 5G /pools/pool1 # /opt/d-cache/bin/dcache pool create 5G /pools/pool2
- Do not use the whole disk space available for the pools! dCache needs some additional space to keep record on metadata linked to files stored in each pool.
Now before dCache can make use of the pool, we have to add it to a dCache domain (usage: /opt/d-cache/bin/dcache pool add [--fqdn] [--domain=<domain>] [--lfs=<mode>] <pool name> <target location>). If no domain is specified, the pool will be added to one named after the hostname (in our case "dcachetogoDomain). lfs=precious makes a pool disk-only, that is without HSM back-end.
# /opt/d-cache/bin/dcache pool add --domain=pool1Domain --lfs=precious pool1 /pools/pool1 # /opt/d-cache/bin/dcache pool add --domain=pool2Domain --lfs=precious pool2 /pools/pool2
For now we need only one directory in dCache's name space to keep things tidy:
# mkdir /pnfs/dcache.org/data/worldwritable
Finally starting dCache
First we make dCache a known service so it will be started during boot phase.
# ln -s /opt/d-cache/bin/dcache /etc/init.d/dcache # chkconfig --add dcache # chkconfig dcache on
Now we are finally prepared to start dCache:
# service dcache start
Usage of /opt/d-cache/bin/dcache
Besides creating new dCache pools and assigning them to dCache domains, this tool is also used for stopping, starting and querying the status of the dCache domains (running on this host).
# service dcache services
- This will list all dCache domains configured for this host.
# service dcache start [service ...]
- This will start all or only the specified dCache services configured for this host.
# service dcache stop [service ...]
- Stop all or only the specified configured dCache domains on this host.
# service dcache restart [service ...]
- restart is just a shortcut for stop followed by start.
File transfer into dCache
There are several clients useable to write into or read from dCache. In this lecture we only demonstrate the native dCache file transfer protocol dcap. As of this protocol is completely unauthorized and our dCache setup has no well defined authorization capabilities – you will learn more about that later – we have to set a dCache pool worldwrite- and readable:
# chmod a+rwx /pnfs/dcache.org/data/worldwritable
Of course normally not root is writing to dCache but any other user. Therefore your image should already know about user ui_user. Switch to him for our user interactions with dCache:
# su - ui_user
Or alternatively at login to your virtual machine
hal9000 login: ui_user <the password is ".grid">
Now we can execute a simple copy into our dCache system using dccp – the dCache cp-command.
$ dccp /bin/bash dcap://<FQDN of your node, where dcap-cell is running>:22125/pnfs/dcache.org/data/worldwritable/dccp.tst
- You will get an error message, but this is expected and the last line of the output is evidence for the successfully completed transfer.
Command failed! Server error message for : "no such file or directory /pnfs/dcache.org/data/worldwritable/dccp.tst" (errno 10001). 616248 bytes in 0 seconds
We just gave dccp a source URL as target definition. Note the same structure like with any http-url:
- <protocol>://<host name>.<domain name>:<port number>/<path to your file>
This way we can also read our just introduced test file:
$ dccp dcap://<FQDN of your node, where dcap-cell is running>:22125/pnfs/dcache.org/data/worldwritable/dccp.tst /tmp/dccp.tst
Make sure the original file and the file read from dCache are not differing:
$ diff /bin/bash /tmp/dccp.tst
It may proof interesting to see what happens in background when using dccp. Repeat the read commandline supplemented with the debug option "-d x.
$ dccp -d 4 dcap://<FQDN of your node, where dcap-cell is running>:22125/pnfs/dcache.org/data/worldwritable/dccp.tst /tmp/dccpdebug.tst
The other tests have to be done as root again.
The Admin Interface
The core mechanism for administrating a dCache instance is the admin interface. This is a service you may connect to using an ssh client. Using the admin interface, you may communicate with the various components making up dCache, query their status and update their behavior. Although dCache is a distributed system, you only ever connect to a single node; dCache will route your messages internally.
# ssh -1 -c blowfish -p 22223 admin@localhost <passphrase is "dickerelke">
Every running dCache service has its own cell, which can be visited in the admin interface. Now we want to see the complete repository of our dCache pool that we used for our dccp transfer:
(local) admin > cd pool1 (pool1) admin >
Now you could view all status information with the command "info" or view all known commands with "help". For our purpose we just execute "ls" for the pool's repository:
(pool1) admin > rep ls
Up till now dCache has no guidelines on where to store which kind of data (this will be explained to you in the next lesson), so our testfile may have been written in both pools we created so far. Thus if you do not get any output from "rep ls", try it again on the cell for the other pool. But before you can change to the other cell, you first have to leave the cell of pool1 by typing in "..":
(pool1) admin > .. (local) admin > cd pool2 (pool2) admin > rep ls
In order to log out you have to return to "local" (with "..") and then you can exit the admin interface with "logoff". Alternatively you can log out directly by pressing CTRL+d.
(pool1) admin > .. (local) admin > logoff
Look forward to the upcoming chapter "Advanced Administration and Tools" which includes the graphical admin interface pcells, among other things.
One of the quickest methods of obtaining simple information about a dCache instance is through the web interface. This provides read-only access to dCache information, such as the pool occupancy or which components are running.
The web interface is provided on port 2288. To see the output, point your favourite web browser at "http://<FQDN of your headnode>:2288/".
You might notice the red text "Module not installed" next to the "statistics" header. In order to activate it, one must first add the service statistics to the SERVICES list inside of the node_config file. In dCacheSetup the variable statisticsLocation should be altered to your situation, and of course the stated directory must exist and be writable. If this is done, run again install.sh and wait for two midnights, dCache needs to create the respective HTML files on hardcoded dates first.
- please contact Xavier Mol from KIT