Details

Written by: andre

Category: Linux Server Admin Guide

Published: 13 March 2013

Hits: 24554

This article describes how to setup a SFTP server installation, plus some advices from a security point of view.
The users will find themselves inside a chrooted environment, and the file system permissions are active:
Every user may have a seperated, isolated environment.

• Article at ipggi.wordpress.com

Step 1: Prepare your system

Create a group for users that are allowed to use sftp, and add some users to it. Since those users communicate with the ssh server without needing a login shell, we will disable shell access for them by setting /bin/false as login shell.

root@testkraxn /home/tester # groupadd sftponly
root@testkraxn /home/tester # cat /etc/group | tail -n 1
sftponly:x:1001:

root@testkraxn /home/tester # useradd garfield -d / -g 1001 -M -N -o -u 1001
root@testkraxn /home/tester # passwd garfield
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully

root@testkraxn ~tester # chsh garfield
Changing the login shell for garfield
Enter the new value, or press ENTER for the default
        Login Shell [/bin/sh]: /bin/false
        
root@testkraxn /home/tester # cat /etc/passwd | tail -n 1
garfield:x:1001:1001::/:/bin/false

Step 2: Configure the SSH Server

root@testkraxn /home/tester # cp /etc/ssh/sshd_config /etc/ssh/sshd_config.`date +%F_%T`.backup
root@testkraxn /home/tester # vim /etc/ssh/sshd_config

replace line:

Subsystem sftp /usr/lib/openssh/sftp-server

with

Subsystem sftp internal-sftp

Then, append these lines at the end of the file:

Match group sftponly
    ChrootDirectory /var/www
    X11Forwarding no
    AllowTcpForwarding no
    ForceCommand internal-sftp

my whole config, to be complete here:

root@killerkraxn ~ # grep -vE '^\s*(#.*|)$' /etc/ssh/sshd_config
Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 768
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp internal-sftp
UsePAM yes
Match group sftponly
    ChrootDirectory /data/sftpusers
    X11Forwarding no
    AllowTcpForwarding no
    ForceCommand internal-sftp

Step 2: Create some test directories

I use a script like this to create a new user: (run as root)

NEW_USER=$1
echo crating user: $NEW_USER

useradd $NEW_USER -d / -G sftponly -M -s /bin/false
groups $NEW_USER

cd /data/sftpusers

echo "setting up home directories..."

mkdir $NEW_USER
chown root:$NEW_USER $NEW_USER
chmod 750 $NEW_USER


mkdir $NEW_USER/readonly
chmod 755 $NEW_USER/readonly
echo "you cannot change this file" > $NEW_USER/readonly/test.txt
chmod 644 $NEW_USER/readonly/test.txt


mkdir $NEW_USER/readwrite
echo "delete me if you wish" > $NEW_USER/readwrite/test.txt
chown $NEW_USER:$NEW_USER -R $NEW_USER/readwrite


mkdir $NEW_USER/no-access
chmod 700 $NEW_USER/no-access
echo "you won't ever read this, douchebag!" > $NEW_USER/no-access/test.txt

echo "you should set a password for the user: passwd $NEW_USER"

If you want to provide access to other folders to the users, you can mount the directory into the users base dir.
In this example, i will grant access to /media/misc/foo to the user garfield:

root@testkraxn ~ # mkdir -p /data/sftpusers/garfield/media
root@testkraxn ~ # mount --bind /media/misc/foo  /data/sftpusers/garfield/media

Finally, the directory structure for the user looks like this:

root@killerkraxn /data/sftpusers # tree
.
└── garfield
    ├── no-access
    │   └── test.txt
    ├── readonly
    │   └── test.txt
    ├── readwrite
    │   └── test.txt
    └── misc
        └── foo --> mountpoint of /media/misc/foo
            └── foo-contents

Restart the ssh server and test the setup:

root@testkraxn ~ # service ssh restart

andre@buenosaires ~ % sftp garfield@ 192.168.56.102
garfield@ 192.168.56.102's password: 
Connected to 192.168.56.102.

sftp> cd /
sftp> pwd
Remote working directory: /

sftp> ls
foo readonly readwrite no-access

sftp> cd no-access
sftp> pwd
Remote working directory: /no-access

sftp> ls
remote readdir("/no-access"): Permission denied

sftp> cd /readwrite
sftp> mkdir bar
sftp> ls
bar test.txt

sftp> cd /readonly
sftp> mkdir bar
Couldn't create directory: Permission denied

sftp> ls
test.txt

sftp> cd /misc
sftp> pwd
Remote working directory: /misc
sftp> ls
foo
  
sftp> exit

And the users do not have access to a shell when trying to connect via ssh:

andre@buenosaires ~ % ssh garfield @192.168.56.102
garfield@ 192.168.56.102's password: 
This service allows sftp connections only.
Connection to 192.168.56.102 closed.

Details

Written by: andre

Category: Linux Server Admin Guide

Published: 27 February 2013

Hits: 22765

My SSH command collection (Collected in the WWW)

1. Enable automatic ssh login using a keyfile

   You might use the ssh-keygen program to create the keyfiles.
   This will enable SSH authentication without needing a password.

   Create a initial ~/.ssh/config file on the client, if not yet present:

   testuser@clienthost $ { echo 'Host *'; echo "IdentityFile ~/.ssh/`whoami`"; } > ~/.ssh/config && chmod 600 ~/.ssh/config

   On CLIENT machine, create private/public key pair, if not yet present.
   (This will create the files ~/.ssh/testuser and ~/.ssh/testuser.pub.)

   testuser@clienthost $ ssh-keygen -t rsa -f ~/.ssh/`whoami` && chmod 600 ~/.ssh/`whoami`

   Install your public key on the REMOTE machine:

   testuser@clienthost $ cat ~/.ssh/`whoami`.pub \
   | ssh remoteuser@remotehost 'cat >> ~/.ssh/authorized_keys'

   Or

   ssh-copy-id remoteuser@remotehost

   Final step, add your private key to your ssh-agent. Otherwise you will get prompted to enter your passphrase every time you try to connect to the remote host.

   ssh-add ~/.ssh/`whoami`

2. Create a tunneled proxy to a remote host

   ssh -N -L9080:localhost:80 theremotehost

   To browse to the remote machine, goto: http://localhost:9080/

3. Connect machine A to machine B through machine C

   Scenario:
   Host A wants to connect to host B on port 443, but they cannot speak directly.
   A third host, C, can access B on port 443, and is able to ssh to host A.
   
   On host C, ssh to host A and create a tunnel that listens on A:1234 and forwards traffic to B:443.

   johndoe@host_C:~$ ssh -N -R 1234:host_B:443 johndoe@host_A

   On host A, localhost:1234 can now be used to access B:443 through the tunnel:

   johndoe@host_A:~$ wget --spider https://localhost:1234 --no-check-certificate 2>&1 | grep 200
   HTTP request sent, awaiting response... 200 OK

4. Setup port forwarding to a remote host

   ssh theuser@remotehost -L 12345:localhost:80

   To access the remote service through the tunnel, visit: http://localhost:12345/
   (This function is not limited to http, every protocol may work)

5. Compare a local file with a remote file

   ssh username@remotehost1 cat /path/to/remote-file | diff /path/to/local-file -

   Useful for checking if there are differences between local and remote files.

6. Mount folder/filesystem through SSH

   sshfs username@remotehost1:/path/to/remotefolder /path/to/localmountpoint

   See sshfs website for installation instructions and documentation.
   This is very convenient, having a remote folder accessible like a local dir.

7. SSH connection through host in the middle

   ssh -t username@remotehost1 ssh endpointhost

   This will chain the ssh connection. Useful when you are not permitted to access the endpointhost from your current host, but from remotehost1.

8. Copy from host1 to host2, through your host

   ssh username1@remotehost1 "cd /somedir/tocopy/ && tar -cf - " | ssh username2@remotehost2 "cd /samedir/tocopyto/ && tar -xf -"

   Good if only you have access to remotehost1 and remotehost2, but they have no access to your host and they have no direct access to each other.

9. Run a GUI tunnelled

   ssh -fX username@remotehost1 xclock

   The SSH server configuration requires:

   X11Forwarding yes

10. Resume scp of a big file

     rsync -partial -progress -rsh=ssh /path/to/localfile username@remotehost1:/path/to/remotefile

    It can resume a broken/interrupted download using rsync.

11. Throttle bandwidth with cstream

    tar -cj /local/file | cstream -t 123k | ssh username@remotehost1 "tar -xj -C /remote/path"

    This copies a compressed folder over the network to username@remotehost1 at 123k bit/s.

Details

Written by: andre

Category: Linux Server Admin Guide

Published: 01 February 2013

Hits: 20749

Create video files

To be compatible to almost any device and os, we will
create three converted versions of the video file using ffmpeg.
a script like this will create them:

    #!/bin/bash

    INPUTFILE="$1"
    if [ "x" = "x$INPUTFILE" ] ; then
        echo error: you need to name an input file
        exit 1
    fi

    OUTPUTFILE="$(basename $INPUTFILE | sed 's/\.[a-z0-9]*$/__converted/g')"
    echo $OUTPUTFILE

    echo
    echo
    echo now converting to MP4 format...
    echo
    avconv \
        -threads 4 \
        -i "$INPUTFILE" \
        -i_qfactor 0.71 \
        -qcomp 0.6 \
        -qmin 10 \
        -qmax 63 \
        -qdiff 4 \
        -trellis 0 \
        -vcodec libx264 \
        -s 720x576 \
        -b:v 8000k \
        -b:a 56k \
        -ar 22050 \
        "$OUTPUTFILE.mp4"

    echo
    echo
    echo now converting to OGV format...
    echo
    avconv \
        -i "$INPUTFILE" \
        -s 720x576 \
        -qmax 63 \
        -b:v 8000k \
        -b:a 56k \
        -ar 22050 \
        -acodec libvorbis \
        "$OUTPUTFILE.ogv"

Put the video onto your website

Then copy the files to the server and embed a videotag like this:

<video id="sampleMovie" width="640" height="360" preload controls>
	<source src="/cms/video.mp4"  type='video/mp4;  codecs="avc1.42E01E, mp4a.40.2"' />
	<source src="/cms/video.ogv"  type='video/ogg;  codecs="theora, vorbis"' />
</video>

Embedded via html5 video tag:

MP4:

OGV:

MP4 + OGV:

Further references:
Very good test page
Article for video conversion
Reference for HTML5 Tags

Details

Written by: andre

Category: Linux Server Admin Guide

Published: 03 December 2012

Hits: 24883

Step 1: Prepare your system

Install subversion, if not yet done:

root@testkraxn ~ # which svn
svn not found
root@testkraxn ~ # apt-get update
root@testkraxn ~ # apt-get install subversion
        The following NEW packages will be installed:
          libapr1 libaprutil1 libdb4.8 libneon27-gnutls libsvn1 subversion
        Do you want to continue [Y/n]? y

Create a single user and run the subversion server as that user. Be sure to make the repository in the filesystem owned by the subversion user as well. From a security point of view, this keeps the repo data well isolated and protected by filesystem permissions, changeable by only the Subversion server process itself.

root@testkraxn ~ # useradd -d /home/subversion -m subversion
root@testkraxn ~ # usermod -a -G subversion subversion
root@testkraxn ~ # chsh subversion # to whatever you prefer

root@testkraxn ~ # cd ~subversion
root@testkraxn ~ # mkdir ~subversion/svn-data
root@testkraxn ~ # chown subversion:subversion -R ~subversion/svn-data

Place a link in the filesystem so we can later access with svn://host:1111/svn/projectX

root@testkraxn ~ # ln -s /home/subversion/svn-data /svn

Step 2: Create a repository

root@testkraxn ~subversion # su subversion
subversion@testkraxn ~ % cd svn-data
subversion@testkraxn ~/svn-data % svnadmin create projectX

Edit your svnserve.conf like this:

subversion@testkraxn ~/svn-data % vim projectX/conf/svnserve.conf
subversion@testkraxn ~/svn-data % grep -vE '^\s*(#.*|)$' projectX/conf/svnserve.conf
[general]
anon-access = none
auth-access = write
password-db = passwd
authz-db = authz

[sasl]

Place your svn users here:

subversion@testkraxn ~/svn-data % vim projectX/conf/passwd
subversion@testkraxn ~/svn-data % grep -vE '^\s*(#.*|)$' projectX/conf/passwd
[users]
andre = asdf27

Step 3: Start and test the svn server daemon process

subversion@testkraxn ~/svn-data % svnserve -d
subversion@testkraxn ~/svn-data % sudo su
root@testkraxn ~ # nmap -sS localhost
PORT     STATE SERVICE
22/tcp   open  ssh
3690/tcp open  svn        <-- you should see this line

Now you should be able to checkout on localhost and from other hosts:

subversion@testkraxn ~/svn-data/projectX % cd /tmp
subversion@testkraxn /tmp % svn co svn://localhost/svn/projectX projectX
subversion@testkraxn /tmp % cd projectX
subversion@testkraxn /tmp/projectX % svn info
Repository Root: svn://localhost/svn/projectX
subversion@testkraxn /tmp/projectX % echo hello > test.txt
subversion@testkraxn /tmp/projectX % svn add test.txt
A         test.txt
subversion@testkraxn /tmp/projectX % svn ci -m "hello world test"
subversion@testkraxn /tmp/projectX % svn update
At revision 1.

or from another host:

anotheruser@anotherhost ~ % svn co svn://vbox0/svn/projectX projectX
A    projectX/test.txt
Checked out revision 1.

Step 4: Configure svn over ssh

This will improve security and convenience, since the users do not need a separate svn user and they can authenticate with a keyfile instead of typing and storing their passwords.

The following requirements must be given:

• The authentication must use the systems unix user acconts. We do not want to manage a new user/password mapping for everybody.
• In the configuration, we need to be able control the following:
  • Put users into groups and enable group based authentication rules.
  • Specify separate permissions for every directory.

Example:

On our system, we have two users:

    tester (the one that has readwrite access)
    leser (the one that cannot commit)

Both accounts are existing on the machine and are able to login via ssh.

Set file system permissions. Users not in group subversion are locked out:

root@testkraxn /etc/ssh # cd ~subversion
root@testkraxn ~subversion # ls -l
drwxrwxr-x 3 subversion subversion 4096 Dec  3 14:58 svn-data/
root@testkraxn ~subversion # chown subversion:subversion -R svn-data # to be sure :D
root@testkraxn ~subversion # chmod o-rwx -R svn-data
root@testkraxn ~subversion # ls -l
drwxrwx--- 3 subversion subversion 4096 Dec  3 14:58 svn-data/

Put users to group "subversion". This will allow to access the svn repository.

root@testkraxn /etc/ssh # usermod -G subversion tester
root@testkraxn /etc/ssh # groups tester
tester : tester subversion

Edit your authz file like this:

subversion@testkraxn ~/svn-data % cat projectX/conf/authz

[aliases]
[groups]
committergroup = tester
readgroup = leser, tester

[/] # relative to project root
# allow nothing by default:
* =
@readgroup = r
@committergroup = rw

[/tags]
* =
committergroup = rw

After this, you should be able to use the svn repo like this:

andrer@caracas ~ svn co svn+ssh://tester@vbox0/svn/projectX/trunk rw
Checked out revision 7.

Optional: Setup websvn viewer for your server

DRAFT

******* install web svn (http://www.websvn.info) ********

root@virtubuntu ~ # apt-get install websvn

or
download bin from 
http://websvn.tigris.org/files/documents/1380/49057/websvn-2.3.3.zip

******* setup test repo ********

root@virtubuntu /var/lib # mkdir svn
root@virtubuntu /var/lib # cd svn
root@virtubuntu /var/lib/svn # svnadmin create helloworld
root@virtubuntu /var/lib/svn # ls -l
drwxr-xr-x 6 tester tester 4096 Sep 12 20:22 helloworld/

root@virtubuntu ~ # dpkg-reconfigure websvn

    Apache configuration:
    [*] apache2

    svn parent repositories:
    /var/lib/svn/helloworld

(apache runs as root in my case, so i do not bother.)

open a browser, and goto:
http://localhost/websvn/

Optional: Migrate SVN projects to avoid permission problems with websvn

If needed with this script you can migrate your projects to the fsfs file system type.

Note on permissions
Due to a limitation in the DB format, the 'svnlook' command needs read-write access to the repository (to create locks etc).
You need to give read-write permissions to the user running your webserver on all your repositories.

Another way of avoiding this problem is by creating SVN repositories with the --fs-type=fsfs option.
Existing DB repositories can be converted to the FSFS format by using the svnadmin dump/load commands.

******* setup in filesystem *********

I want to convert all projects that are in directory "public".
The target directory is "migration-stage"

andre@killerkraxn /data/subversion/svn-data/andre % ls -l
drwx------  5 andre andre 4096 2013-07-30 18:41 private/
drwxrwxr-x 36 andre andre 4096 2013-02-13 18:54 public/
andre@killerkraxn /data/subversion/svn-data/andre % ./migrate.sh public  # see below


******* migrate.sh script *********

#!/bin/bash

sourceRootDir="$1"                                                             
migratedRootDir="migration-stage"
mkdir -p "$migratedRootDir"

find "$sourceRootDir" -type d -mindepth 1 -maxdepth 1 \
| sort \
| while read sourceRepo
do
        migratedRepo="${migratedRootDir}${sourceRepo#$sourceRootDir}"
        dumpFile="$migratedRepo.dump"

        echo "migrating repository $sourceRepo to $migratedRepo ..."

        echo "1. create new repo with correct fs type: $migratedRepo"
        svnadmin create --fs-type=fsfs "$migratedRepo"

        echo "2. create svn dump to: $dumpFile"
        svnadmin dump "$sourceRepo" > "$dumpFile"

        echo "3. drop the dump in the new repo"
        cat "$dumpFile" | svnadmin load "$migratedRepo"

        echo "done with migration of $sourceRepo"; echo
done



******** example output *********

migrating repository public/web-jsf2-getstarted to migration-stage/web-jsf2-getstarted ...
1. create new repo: migration-stage/web-jsf2-getstarted

2. create svn dump to: migration-stage/web-jsf2-getstarted.dump
* Dumped revision 0.
* Dumped revision 1.

      [...]

* Dumped revision 11.
* Dumped revision 12.

3. drop the dump in the new repo
<<< Started new transaction, based on original revision 1
     * adding path : branches ... done.
     * adding path : tags ... done.
     * adding path : trunk ... done.
------- Committed revision 1 >>>
<<< Started new transaction, based on original revision 2
     * editing path : trunk ... done.
     * adding path : trunk/pom.xml ... done.
     * adding path : trunk/web ... done.
     * adding path : trunk/web/WEB-INF ... done.
     * adding path : trunk/web/WEB-INF/web.xml ... done.
------- Committed revision 2 >>>
<<< Started new transaction, based on original revision 3

      [...]

<<< Started new transaction, based on original revision 11
     * editing path : trunk/pom.xml ... done.
------- Committed revision 11 >>>
<<< Started new transaction, based on original revision 12
     * editing path : trunk/web/WEB-INF/web.xml ... done.
------- Committed revision 12 >>>
done with repo public/web-jsf2-getstarted