Marcin's Developer Notes - notes and pro tips for Linux

Bash

For loop

for file in $(ls)
do
   echo "Asdf "$file
done

Fix last command in editor

Opens last command in editor (Vim / Nano) and executes on exit.

fc $!

xargs

Basic usage

xargs allows to take lines from standard input and execute command for each line, i.e:

docker ps -q | xargs docker kill

will kill all running containers.

Custom parameter position

Sometimes you might want to adjust the position of the argument from standard input. You can do this like this:

ls | xargs -I@ cp @ ./destination_folder

Printing the commands

If you want to print the commands that are executed, use -t param:

cat file.txt | xargs -t -i echo -e {}"\n"
# output:
# echo -e test\n
# test
# 
# echo -e 1\n
# 1
# 
# echo -e 2\n
# 2
# 
# echo -e 3\n
# 3

xargs with `find` when file names are werid (i.e containing aprostrophes etc.)

find ~/dataset/train -type f | cat | xargs -d$'\n' -P 8 -n 1 -I@ cp "@" .

Alternative: If you encounter a similar problems the solutions is to add -printf ‘"%p”\n’ to the find command line that will enclose all file names into double quotes:

find . -type f -printf '"%p"\n' | xargs grep string

echo

echo with new line:

echo -e "MyString\n"

find

find with file size range

# finds files in size between 1KB and 128KB, can use M or G units too
find . -type f -size +1k -size -128k

find files not matching extensions

find . -not -name "*.jpg" -not -name "*.png" -type f

netcat

listen on UDP port, can emulate Logstash:

nc -kluvw 1 localhost 5514

File operations

Rename file by replacing string

Version 1

rename 's/WHAT_TO_REPLACE/SUBSTITUTION/' *

Version 2

for file in AreYouOnTheSquare*; do mv $file "${file#AreYouOnTheSquare}"; done

Sum file size of files in folder

du -h *.csv | cut -f1 | sed 's/M//' | paste -sd+ | bc

Remove apostrophes from file names

Apostrophes can mess up xargs usage.

for i in ./*; do mv "$i" "$(echo "$i" | tr -d "'")"; done

ls sort by size

ls --sort=size -lh

Check if file exists singleline

(test -f /path/to/file && echo "Exists") || echo "Not exists"

Obfuscate file names (rename to random)

ls | while read file; do mv "$file" "$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 16 | head -n 1).<extension>"; done

File converting

Convert flac to mp3 in terminal

Converts all files in folder:

sudo apt-get install flac
sudo apt-get install lame
for f in *.flac; do flac -cd "$f" | lame -b 320 - "${f%.*}".mp3; done

Processes

Monitoring memory usage

while true; do
        sudo pmap -x `ps aux | grep main.py | awk '{print $2}' | head -1` | tail -1 >> memory.txt
        sleep 1
done

Monitoring network usage (htop for network)

Use nload tool.

sudo apt-get install -y nload

Show processes without `ps` command

Useful e.g. for docker containers / kubernetes

ls -l /proc/*/exe

Kill processes without `ps` command available

kill -9 $(ls -l /proc/*/exe | grep python | cut -d'/' -f3)

tmux

Split windows

Ctrl+b, Ctrl+"

Ctrl+b, Ctrl + %

Scroll window

Ctrl+b, [ then up/down or Page Up/Page Down

grep

Return only matches

Return only matches of regex:

grep -o -e "2019-.*,"

Inverse match (grep reverse)

Returns all lines not ending with either / or ::

grep -v -E "(/:)$"

Remove empty lines

Not grep, but useful with the inverse match (see above):

grep -v -E "(/:)$" | awk NF

Apply actual regex (Perl based)

Useful when working with patterns such as \d etc.

cat file.txt | grep -P "\d+"

Match multiple patterns at once

(and output them back into single line)

Example file logs3.txt:

[21/Feb/2022:07:04:26 +0000] 31.0.38.54 POST HTTP/1.1 upstream_response_time 0.352
[21/Feb/2022:09:31:03 +0000] 37.47.144.90 POST HTTP/1.1 upstream_response_time 0.348
[21/Feb/2022:07:04:02 +0000] 31.0.38.54 POST HTTP/1.1 upstream_response_time 0.352
[21/Feb/2022:09:30:30 +0000] 37.47.144.90 POST HTTP/1.1 upstream_response_time 0.352
[21/Feb/2022:09:31:31 +0000] 37.47.144.90 POST HTTP/1.1 upstream_response_time 0.348
[21/Feb/2022:06:43:16 +0000] 37.248.219.14 POST HTTP/1.1 upstream_response_time 1.096
[21/Feb/2022:09:30:27 +0000] 37.47.144.90 POST HTTP/1.1 upstream_response_time 0.352
[21/Feb/2022:09:39:36 +0000] 37.47.144.90 POST HTTP/1.1 upstream_response_time 0.348
[21/Feb/2022:09:39:36 +0000] 37.47.144.90 POST HTTP/1.1 upstream_response_time 0.348
[21/Feb/2022:07:05:58 +0000] 31.0.38.54 POST HTTP/1.1 upstream_response_time 0.400

cat logs3.txt | grep -P -o "(\d\d/\w\w\w/\d\d\d\d)|(\d+\.\d+\.\d+.\d+)" | paste -sd ' \n'

Outputs:

21/Feb/2022 31.0.38.54
21/Feb/2022 37.47.144.90
21/Feb/2022 31.0.38.54
21/Feb/2022 37.47.144.90
21/Feb/2022 37.47.144.90
21/Feb/2022 37.248.219.14
21/Feb/2022 37.47.144.90
21/Feb/2022 37.47.144.90
21/Feb/2022 37.47.144.90
21/Feb/2022 31.0.38.54

curl

download file with auto name

curl -O -J <URL>

jq

Filter by string

cat file.json | jq '.[] | select(.type == "GHOST")| {id: .id, type: .type}'

No matches...