Dealing with files and mountpoints

1. Compressing Archiving files

List contents of zip archive

unzip -l FASTQ.files.zip

Zip directory contents

zip -r -2 TCGA\ SKCM.zip TCGA\ SKCM/ -x "*.DS_Store"

GZip extract file while keeping the original archive

gzip -kd File1.gz File2.gz

Zip folders to server

zip -r -2 /Volumes/compbiomed/Server_Address/file.zip Local_directory/

Heavy Compression and Process only changed files and folders

zip -r -f -9 /Volumes/compbiomed/Server_Address/file.zip Local_directory/

Heavy compression using XZ

brew install xz
tar -vcf SAM.Files.tar *.sam
xz -v5k SAM.Files.tar

Heavy compression in one command to save tarring disk space.

tar -cf - Folder1 | xz -v5 -T4 -z - > Folder1.tar.xz

tar -cf - Folder1 | xz -vz - > Folder1.tar.xz

Here -T4 allocates 4 CPU threads to XZ compression making it faster.

Here - is the STDIO so tar sends the tar to ‘-’ and xz reads the input from ‘-’. This operation is dependent on the IO speed of the storage.

Compressing more than one file (not entire folder, but a few files only)

tar -cf - file1.txt file2.txt | xz -v5 - > test.tar.xz

2. File operations

Compare two files:

cmp File1.vcf ../File2.vcf && echo 'Same' || echo 'Different'

git diff --no-index File1 File2

Concatenate two files

cat File.ToAppend.txt >> WhatToAppend.File.txt

To merge several files to create a new file use:

cat file1 file2 file3 > newfile

To merge several files into a single existing file use:

cat file2 file3 file4 >> file1

Transfer ownership of some files from root to a user

whoami # Chech your username (correct spelling)
id # Generates output with all groups you are  apart of.
chown -v User:Group FileName.ext

To selectively copy files from one directory to another.

Remember that this also copies the directory structure required to copy the files.

rsync -avm --include='*.bamcount.tsv' -f 'hide,! */' . ../HiDP.Analysis/Bamcount

Watch size of a file change over time

brew install watch # if you don't have watch on OSX
watch -n 5 "du -h FileName.txt"

Temporarily enable write permissions on a directory

chomod 777 /path/to/dir # open access rights to everyone
chmod 775 /path/to/dir # close access rights

List storage occupied by folders in a directory

du -h -d 1

# Sort output based on size
du -h -d 1 | sort -h

Check which process is using which file

Very helpful when unmounting a drive generates a warning that “disk is being used” or something like that.

lsof | rg "DriveName"

3. Mounting storage devices

Mounting network media using SSHFS in mac True10-01

sshfs -p 22 dchakrob@portal.btk.fi:/ ~/fmsc_mount/,oauto_cache,follow_symlinks,reconnect,defer_permissions,no_readahead,noappledouble,nolocalcaches,volname=FMSCmnt

Mount BTK network drive

sshfs -p 22 dchakrob@portal.btk.fi:/home/dchakrob ~/fmsc_mount/ -oauto_cache,follow_symlinks,reconnect,defer_permissions,no_readahead,noappledouble,nolocalcaches,volname=FMSCmnt

Mount / (root) of UTU iMAC hard drive

sshfs -p 22 Deepankar@dyn59-155.med.net.utu.fi:/ ~/UTU.iMac/ -oauto_cache,follow_symlinks,reconnect,defer_permissions,no_readahead,noappledouble,nolocalcaches,volname=UTU.iMac

Mount MY USER of UTU iMAC hard drive

sshfs -p 22 Deepankar@dyn59-155.med.net.utu.fi:/Users/Deepankar/ ~/UTU.iMac/ -oauto_cache,follow_symlinks,reconnect,defer_permissions,no_readahead,noappledouble,nolocalcaches,volname=UTU.iMac

Unmounting a network drive on mac

umount ~/fmsc_mount
# force unmount
umount -f ~/fmsc_mount/

Mount NTFS drive on MacOS

# Install from homebrew
brew install ntfs-3g

sudo umount /dev/diskNumber
sudo ntfs-3g /dev/DiskNumber ~/Vols/Path -olocal -oallow_other