Dealing with files and mountpoints
1. Compressing Archiving files
List contents of zip archive
unzip -l FASTQ.files.zip
Zip directory contents
zip -r -2 TCGA\ SKCM.zip TCGA\ SKCM/ -x "*.DS_Store"
GZip extract file while keeping the original archive
gzip -kd File1.gz File2.gz
Zip folders to server
zip -r -2 /Volumes/compbiomed/Server_Address/file.zip Local_directory/
Heavy Compression and Process only changed files and folders
zip -r -f -9 /Volumes/compbiomed/Server_Address/file.zip Local_directory/
Heavy compression using XZ
brew install xz
tar -vcf SAM.Files.tar *.sam
xz -v5k SAM.Files.tar
Heavy compression in one command to save tarring disk space.
tar -cf - Folder1 | xz -v5 -T4 -z - > Folder1.tar.xz
tar -cf - Folder1 | xz -vz - > Folder1.tar.xz
Here -T4
allocates 4 CPU threads to XZ compression making it faster.
Here -
is the STDIO so tar sends the tar to ‘-’ and xz reads the input from ‘-’. This operation is dependent on the IO speed of the storage.
Compressing more than one file (not entire folder, but a few files only)
tar -cf - file1.txt file2.txt | xz -v5 - > test.tar.xz
2. File operations
Compare two files:
cmp File1.vcf ../File2.vcf && echo 'Same' || echo 'Different'
git diff --no-index File1 File2
Concatenate two files
cat File.ToAppend.txt >> WhatToAppend.File.txt
To merge several files to create a new file use:
cat file1 file2 file3 > newfile
To merge several files into a single existing file use:
cat file2 file3 file4 >> file1
Transfer ownership of some files from root to a user
whoami # Chech your username (correct spelling)
id # Generates output with all groups you are apart of.
chown -v User:Group FileName.ext
To selectively copy files from one directory to another.
Remember that this also copies the directory structure required to copy the files.
rsync -avm --include='*.bamcount.tsv' -f 'hide,! */' . ../HiDP.Analysis/Bamcount
Watch size of a file change over time
brew install watch # if you don't have watch on OSX
watch -n 5 "du -h FileName.txt"
Temporarily enable write permissions on a directory
chomod 777 /path/to/dir # open access rights to everyone
chmod 775 /path/to/dir # close access rights
List storage occupied by folders in a directory
du -h -d 1
# Sort output based on size
du -h -d 1 | sort -h
Check which process is using which file
Very helpful when unmounting a drive generates a warning that “disk is being used” or something like that.
lsof | rg "DriveName"
3. Mounting storage devices
Mounting network media using SSHFS in mac True10-01
sshfs -p 22 [email protected]:/ ~/fmsc_mount/,oauto_cache,follow_symlinks,reconnect,defer_permissions,no_readahead,noappledouble,nolocalcaches,volname=FMSCmnt
Mount BTK network drive
sshfs -p 22 [email protected]:/home/dchakrob ~/fmsc_mount/ -oauto_cache,follow_symlinks,reconnect,defer_permissions,no_readahead,noappledouble,nolocalcaches,volname=FMSCmnt
Mount / (root) of UTU iMAC hard drive
sshfs -p 22 [email protected]:/ ~/UTU.iMac/ -oauto_cache,follow_symlinks,reconnect,defer_permissions,no_readahead,noappledouble,nolocalcaches,volname=UTU.iMac
Mount MY USER of UTU iMAC hard drive
sshfs -p 22 [email protected]:/Users/Deepankar/ ~/UTU.iMac/ -oauto_cache,follow_symlinks,reconnect,defer_permissions,no_readahead,noappledouble,nolocalcaches,volname=UTU.iMac
Unmounting a network drive on mac
umount ~/fmsc_mount
# force unmount
umount -f ~/fmsc_mount/
Mount NTFS drive on MacOS
# Install from homebrew
brew install ntfs-3g
sudo umount /dev/diskNumber
sudo ntfs-3g /dev/DiskNumber ~/Vols/Path -olocal -oallow_other