Damien Gonot
Home Blog Notes About

UNIX Commands

Homepage / Notes / Computer Science / UNIX Commands

UNIX Commands

rm

ReMoves a file

rm ~/tmp/unix_example

echo

Writes argument to new line of file

echo "yo" >> ~/tmp/unix_example

cat

Display the content of specified file

cat ~/tmp/unix_example
yo

sort

cat ~/tmp/sort_example
8
4
6
2
3
1
5
9
7
sort ~/tmp/sort_example
1
2
3
4
5
6
7
8
9

Works with pipe operator too:

cat ~/tmp/sort_example | sort

uniq

cat ~/tmp/uniq_example
2
1
3
3
2
2
2
1
cat ~/tmp/uniq_example | uniq
2
1
3
2
1
cat ~/tmp/uniq_example | sort | uniq
1
2
3

tail

Display the last part of a file

cat ~/tmp/sort_example | sort | tail -n 5
5
6
7
8
9

tail -f file: keep reading file until Ctrl-C

less

less file: interactively read file

less -F file equivalent to tail -f file

grep

Find patterns in files

cat ~/tmp/csv_example
productidproductnameurlpricecurrency
1Nike shoesexample.com/nike80EUR
2Adidas shoesexample.com/adidas75EUR
3Puma shoesexample.com/puma82EUR
4Burton bagexample.com/burton55EUR
5Canada Goose parkaexample.com/canada-goose600CAD
cat ~/tmp/csv_example | grep "Nike"
1Nike shoesexample.com/nike80EUR

awk

https://www.gnu.org/software/gawk/manual/gawk.html https://earthly.dev/blog/awk-examples/ https://www.cs.princeton.edu/courses/archive/spring19/cos333/awk.help

cat ~/tmp/csv_example | awk -F ',' '{print $2}'
product_name
Nike shoes
Adidas shoes
Puma shoes
Burton bag
Canada Goose parka
cat ~/tmp/csv_example | awk -F ',' '{print $4}'
price
80
75
82
55
600

sed

sed 's/example/amazon/' ~/tmp/csv_example
productidproductnameurlpricecurrency
1Nike shoesamazon.com/nike80EUR
2Adidas shoesamazon.com/adidas75EUR
3Puma shoesamazon.com/puma82EUR
4Burton bagamazon.com/burton55EUR
5Canada Goose parkaamazon.com/canada-goose600CAD

cut

cat ~/tmp/csv_example | cut -d',' -f3
url
example.com/nike
example.com/adidas
example.com/puma
example.com/burton
example.com/canada-goose

tr

cat ~/tmp/csv_example | cut -d',' -f3 | tr / ?
url
example.com?nike
example.com?adidas
example.com?puma
example.com?burton
example.com?canada-goose

ncdu

Faster implementation for SSDs, written in Go: gdu

reset

Reset comsol to its initial state

jq

https://stedolan.github.io/jq/manual/ jq file.json to parse JSON file

Pretty printing

echo '{"key1":{"key2":"value1"}}' | jq '.'
{
  "key1": {
    "key2": "value1"
  }
}

Selecting a specific key

curl https://api.github.com/repos/mewfree/mewfree | jq '.owner'
{
  "login": "mewfree",
  "id": 8095395,
  "node_id": "MDQ6VXNlcjgwOTUzOTU=",
  "avatar_url": "https://avatars.githubusercontent.com/u/8095395?v=4",
  "gravatar_id": "",
  "url": "https://api.github.com/users/mewfree",
  "html_url": "https://github.com/mewfree",
  "followers_url": "https://api.github.com/users/mewfree/followers",
  "following_url": "https://api.github.com/users/mewfree/following{/other_user}",
  "gists_url": "https://api.github.com/users/mewfree/gists{/gist_id}",
  "starred_url": "https://api.github.com/users/mewfree/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/mewfree/subscriptions",
  "organizations_url": "https://api.github.com/users/mewfree/orgs",
  "repos_url": "https://api.github.com/users/mewfree/repos",
  "events_url": "https://api.github.com/users/mewfree/events{/privacy}",
  "received_events_url": "https://api.github.com/users/mewfree/received_events",
  "type": "User",
  "site_admin": false
}
curl https://api.github.com/repos/mewfree/mewfree | jq '.owner.html_url'
"https://github.com/mewfree"

Selecting elements from an array

echo "[1,2,3,4,5]" | jq '.[0]'
1
echo "[1,2,3,4,5]" | jq '.[0:2]'
[
  1,
  2
]
echo "[1,2,3,4,5]" | jq '.[2:]'
[
  3,
  4,
  5
]
echo "[1,2,3,4,5]" | jq '.[-2:]'
[
  4,
  5
]

Selecting keys from elements from an array

echo '[{"title": "a"}, {"title": "b"}, {"title": "c"}]' | jq '.[].title'
"a"
"b"
"c"
echo '[{"title": "a"}, {"title": "b"}, {"title": "c"}]' | jq '.[0].title'
"a"

Raw strings

echo '[{"title": "a"}, {"title": "b"}, {"title": "c"}]' | jq -r '.[0].title'
a

Join strings

echo '[{"title": "a"}, {"title": "b"}, {"title": "c"}]' | jq -j '.[].title'
abc

Array constructor

echo '[{"title": "a"}, {"title": "b"}, {"title": "c"}]' | jq '[ .[].title ]'
[
  "a",
  "b",
  "c"
]

Select multiple fields

By iterating:

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq '.[].type, .[].title'
"movie"
"book"
"tv show"
"The Social Network"
"Ready Player One"
"Halt And Catch Fire"

Using the pipe operator:

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq '.[] | .type, .title'
"movie"
"The Social Network"
"book"
"Ready Player One"
"tv show"
"Halt And Catch Fire"

Pipe operator wrapped in array constructor:

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq '[ .[] | .type, .title ]'
[
  "movie",
  "The Social Network",
  "book",
  "Ready Player One",
  "tv show",
  "Halt And Catch Fire"
]

Using the object constructor:

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq '[ .[] | {media_type: .type, title: .title } ]'
[
  {
    "media_type": "movie",
    "title": "The Social Network"
  },
  {
    "media_type": "book",
    "title": "Ready Player One"
  },
  {
    "media_type": "tv show",
    "title": "Halt And Catch Fire"
  }
]

Sort / Reverse / Length

echo '["3","2","1"]' | jq 'sort'
[
  "1",
  "2",
  "3"
]
echo '["1","2","3"]' | jq 'reverse'
[
  "3",
  "2",
  "1"
]
echo '["1","2","3"]' | jq 'length'
3

Map / Select

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq 'map({ title: .title })'
[
  {
    "title": "The Social Network"
  },
  {
    "title": "Ready Player One"
  },
  {
    "title": "Halt And Catch Fire"
  }
]
echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq 'map(select(.type == "tv show")) | map({title: .title})'
[
  {
    "title": "Halt And Catch Fire"
  }
]
echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq 'map(select(.type != "book")) | map({title: .title, date: .release_date})'
[
  {
    "title": "The Social Network",
    "date": "2010"
  },
  {
    "title": "Halt And Catch Fire",
    "date": "2014"
  }
]

fx

fx file.json to view and interact with JSON file

z

Alternative cd? https://github.com/rupa/z

tree

Display directory structure

tree -I .git ignore .git tree -a show "hidden" files

stow

https://alexpearce.me/2016/02/managing-dotfiles-with-stow/ stow vim creates a symlink between files in vim and upper directory

shuf

"shuffle"

seq 10 | shuf
9
6
7
1
10
5
8
3
2
4
seq 10 | shuf -n 1
9

dasel

https://github.com/TomWright/dasel jq for JSON, YAML, TOML, XML and CSV

Querying for a specific field

echo '{"id": 1, "name": "Damien"}' | dasel -r json '.name'
"Damien"

JSON => CSV

echo '{"id": 1, "name": "Damien"}' | dasel -r json -w csv
id,name
1,Damien

CSV => JSON

printf 'id,name\n1,Damien\n2,Emilie' | dasel -r csv -w json
{
  "id": "1",
  "name": "Damien"
}
{
  "id": "2",
  "name": "Emilie"
}

Miller

https://miller.readthedocs.io/en/latest/index.html Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON. You get to work with your data using named fields, without needing to count positional column indices.

"cat" (unmodified)

printf 'id,name\n1,Damien\n2,Emilie' | mlr --csv cat
id,name
1,Damien
2,Emilie

Pretty print

printf 'id,name\n1,Damien\n2,Emilie' | mlr --csv --opprint cat
id name
1  Damien
2  Emilie

Select column(s)

printf 'id,name\n1,Damien\n2,Emilie' | mlr --csv cut -f name
name
Damien
Emilie

htmlq

https://github.com/mgdm/htmlq

curl --silent https://www.damiengonot.com/blog | htmlq --text article h3 a
🧨 Supercharge Your Spreadsheets With Free Emailing 📮
How to Automatically Pull Facebook Ads Data in Google Spreadsheet?
Guide to Facebook Insights API (Part 2)
Guide to Facebook Insights API
Getting Started with Facebook Marketing API
Connecting Google Ads Scripts with Google Spreadsheets
Google Ads Scripts: Programmatically Create Ads
Google Ads Scripts: Modify Entities
Google Ads Scripts: Reading Data
A Brief Introduction to Google Ads Scripts

pup

https://github.com/ericchiang/pup

curl --silent https://www.damiengonot.com/blog | pup 'article h3 a text{}'
🧨 Supercharge Your Spreadsheets With Free Emailing 📮
How to Automatically Pull Facebook Ads Data in Google Spreadsheet?
Guide to Facebook Insights API (Part 2)
Guide to Facebook Insights API
Getting Started with Facebook Marketing API
Connecting Google Ads Scripts with Google Spreadsheets
Google Ads Scripts: Programmatically Create Ads
Google Ads Scripts: Modify Entities
Google Ads Scripts: Reading Data
A Brief Introduction to Google Ads Scripts

No support for emojis?

ufw

"UncomplicatedFireWall"

/!\ allow SSH when setting up ufw on a remote server so you don't lose the connection sudo ufw allow ssh

Status: sudo ufw status Enable: sudo ufw enable

Deny incoming but allow outgoing: sudo ufw default deny incoming sudo ufw default allow outgoing

csvquote

https://github.com/dbro/csvquote Enables common unix utlities like cut, awk, wc, head to work correctly with csv data containing delimiters and newlines

rsync

rsync {origin} {destination}

Works over ssh too: rsync [email protected]:/home/user /home/user/backup

Common arguments:

ffmpeg

Crop video

Live Preview

Crop bottom 100px from source.mp4: ffplay -i source.mp4 -vf "crop=in_w:in_h-100:0:0"

Actual

ffmpeg -i source.mp4 -filter:v "crop=in_w:in_h-100:0:0" -c:a copy destination.mp4

Help

Traditionally, can use man {command} to get info about command but can be quite lengthy and lack examples. tldr {command} solves this problem by being a short cheatsheet with many examples.

Resources

https://earthly.dev/blog/command-line-tools/ https://news.ycombinator.com/item?id=27992073