Some notes regarding UNIX Systems. More notes can be found here:

UNIX Program Execution, CPU, Memory and Virtualization

 

 

Operating System

UNIX began 1970s ATT Bell Labs. Many variations of Unix today. 

UNIX Architecture:

• Kernel
• System Calls
• Shell and Library routines
• Applications

UNIX documentation part of OS – man pages. (manual pages). Man page contains:

• Name
• Synopsis (how to use)
• Description
• Options
• Environment

 

 

File System

Linux file system

• File system used is a choice with Linux
  • EXT 2/3/4 = Extended File System 2/3/4 for Linux 1992,2008 (ext4)
    • Ext4 has max file 16TB and volumes 1EB
    • No native Windows/MacOS support
  • XFS
  • Btrfs
  • ZFS = Zed File System
    • Originally by Sun Microsystems, today OpenZFS
    • Used by Linux, FreeBSD, 

 

 

 

• File system used in other Operating Systems
  • HFS = Hierarchical File System for MacOS
    • 1998, HSF+ (extended) files and volumes up to 8EB
    • 2017, APFS Apple file system
    • No native Windows/Linux support
  • FAT = File Allocation Table (Windows) 
    • Had lots of wasted space (Slack)
    • FAT32 still used a lot in USB
    • EXFAT = Extended FAT 
      • For high capacity SD cards, not always compatible with some FAT
      • Max file size is 16EB exabytes, therefore standard for SDXC
  • NTFS = New Technology File System 1993
    • Much larger capacity than FAT (up to 16 EB exabytes)
    • Much more features too like encryption

Inode data elements list

• File size
• User ID (UID)
• Group ID (GID)
• Mode (rwx, special flags)
• Timestamps (ctime, atime, mtime)
• Link count
• Pointers to data blocks
• Many inode elements are set by POSIX

POSIX

The Portable Operating System Interface (POSIX) is a family of standards specified by the IEEE Computer Society for maintaining compatibility between operating systems. POSIX defines both the system- and user-level application programming interfaces (API), along with command line shells and utility interfaces, for software compatibility (portability) with variants of Unix and other operating systems. POSIX is also a trademark of the IEEE. POSIX is intended to be used by both application and system developers.

 

Unix has hierarchical file system

• Tree structure
• Starts with root directory (which is file itself named “/”)
• All files organized below /

Unix uses virtual layer design

• To the logical layer, file system appears as single tree, even though many devices beneath
• Physical layer often spans many storage devices

Types of UNIX files

• Directory
• Regular 
• Symbolic links – generalizes file names
• Pipes
• Sockets
• Special – for I/O devices

Unix Directory

• Directory is a file of directory entries
• Logically a directory entry is a pair of filename and file attributes
• Attributes are:
  • Type of file
  • Size of file
  • Potentially much more info depending on design of file system

Filename

• Names in a directory are filenames
• UNIX filename may not contain characters / (slash) or null
• Slash separates filename, forms pathname
• NULL terminates a pathname
• Almost all current versions of UNIX allow filenames up to 255 char
• For portability use only letters, digits, period, dash, and underscore
• Every directory contains the filenames . (called dot) and .. (dot-dot)
  • . = directory itself
  • .. = parent directory

Pathname

• Sequence of filenames with slashes
• Absolute Pathname = starts with /
• Relative pathname = not starts with /
  • Refers to files relative to current directory

Some important UNIX directories:

• / = root
• /etc = UNIX configuration files
• /etc/passwd = user information
• /etc/groups = group info
• /etc/rc* = initialization scripts for services
• /dev = devices attached
• /usr = libraries and tools
• /usr/bin = binary forms of tools
• /usr/lib = libraries needed by tools
• /usr/include = include files for libraries
• /home/username = user home directory

Symbolic Links / Soft Links

• Symbolic link/soft link = file that contains reference to another file or directory – called target
• Symbolic links can point outside a partition
• Form is (file name, i-node number and path to file)
  • Can be absolute or relative
• Symbolic link can point to nothing (invalid)
  • If target is removed
  • Becomes a dangling symbolic link

 

Exploring a file system / Tools

• fdisk & gdisk – manipulate disk partitions
• mkfs – build a Linux file system
• tune2fs – adjust filesystem parameters
• cryptsetup luksDump – show header info
• ls -ia – show inode numbers
• stat – display file(system) status

 

Types of files

• Directories
  • Lists of other files
• Special files
  • Mechanisms for input/output
  • Often in /dev
• Links
  • Symbolic links
  • Hard links
• (Domain) Sockets
  • Inter-process networking protected by file system access controls
• Named pipes
  • Similar to sockets, without networking semantics
• Regular files

 

ls -l command

• Meaning of 1st character in each line from ls –l
  • Symbol | Meaning
  • – Regular file
  • d Directory
  • l Symbolic link
  • c Special file
  • s Socket
  • p Named pipe
  • b Block device

chmod – setting file mode

• chmod – CHange file MODe bits
  • Symbolic mode descriptors – ugoa[+-]rwxXst
  • Numeric mode descriptors
    • read = 4 (0b100)
    • write = 2 (0b010)
    • execute = 1 (0b001)
    • Example: chmod 0711 myfile = rwx–x–x
  • 4000 for setuid and none of read, write, and execute for owner, group, and everyone
  • 2000 for setgid …
  • 1000 for sticky

setuid/setgid bits

• setuid – set user ID
  • when executed, the process runs with the uid of the file owner
• setgid – same idea as setuid, but with gid
  • Directories are an exception; files created within a setgid directory inherit the group ID of that directory

sticky bit

• When a directory’s sticky bit is set, the file system treats files in that directory differently
  • Regardless of write and execute permissions, only the file owner, directory owner, or root user can rename or delete files in that directory
• Sticky bit typically set on directory /tmp to prevent users from moving or deleting other users’ temporary files

chmod examples for uid, gid, and sticky bits

• Chmod 4700 /usr/bin/vim
  • Or chmod u+s /usr/bin/vim (user + sticky bit)
• Chmod 2700 /usr/bin/vim
  • Or chmod g+s /usr/bin/vim (group + sticky bit)
• Chmod 1755 /tmp
  • Or chmod +t /tmp

Extended attributes

• For linux system with this feature, extension to the normal attributes associated with every inode
• Name:value pairs associated with files
• getfattr – get extended attributes of filesystem objects

Discretionary access control (user controlled permissions)

• User managed
  • classic rwx u/g/a file permissions
• Finer grained – Access Control Lists (ACLs)
• More flexible than owner/group/everyone permissions categories
• setfacl – set file access control list
• getfacl – get file access control lists
• POSIX has access control lists (ACLs)

setfacl & getfacl examples using ACL

• Example
  • setfacl -m u:apache:r /some/path
    • Modifies ALC to give user “apache” read permission for file /some/path
    • On the file permission it will show a “+” at the end
  • getfacl /some/path
    • Prints the ACL for /some/path

 

Volatility of storage technology

• Storage tech requires electrical energy
• When no electrical, no access to data
• When power restored:
  • Volatile storage tech will have lost stored data
  • Non-Volatile storage tech will have retained data

Volatile storage tech, from fastest to slowest

• Processor registers (flip-flops)
• Processor cache (SRAM, static RAM) Random Access Memory
• Computer main memory (DRAM, Dynamic RAM)

Non-Volatile storage tech, from fastest to slowest access time

• Flash memory
• Hard disk drive
• CD/DVD
• Tape
• Cloud storage

 

 

 

 

User Permissions

UNIX was designed to be multiuser

• Database of users is in the /etc/passwd
• Example: cat /etc/passwd | grep grr
  • grr:x:759:759:Gustavo rivera,,,:/homes/grr /bin/tcsh
  • login:userid:groupid:Name,,,:homedir:shell
• Every user has a unique USER ID
  

Root user

• Special user with special privileges
• Only root can modify files anywhere in system
• Only root can add users, reset password
• To login to root, aka superuser, use the command su (substitute user)

Groups

• Each user also has GROUP ID
• Used to facilitate file sharing, limiting access for non-group members
• One user may be in many groups
• /etc/group describes the groups in system

File Attributes

• Attributes provide useful metadata about files
• Typical attributes:
  • Ownership
  • Permissions
  • Date and time
  • Number links
  • File size
  • Other (extended) attributes

File owner classes

• User = owner of the file
• Group = collection of users
• Other = a user who is neither the file owner nor a group member

File permissions

• Read
• Write
• Execute

-rw-rw-r-- (10 characters)
DirectoryBit,User,Group,Other (groups of 3 bits)

Example

• Readable and writable by user and group, readable by others:
  • chmod u+rw hello.txt
  • chmod g+rw hello.txt
  • chmod o+r hello.txt
  • chmod o-w hello.txt
• Allow executable by all:
  • chmod ugo+x hello.sh
• Use Base8 / Octal notations:
  • chmod 664 hello.c
  • User bits = 110 (6)
  • Group bits = 110 (6)
  • Others bits = 100 (4)

Directory bit

• When file is directory, sets the directory bit 
• Directories have permissions
• With +x and -r permissions, the files within directory are accessible but invisibles

 

 

Unix Processes

Program is an executable file residing on disk in directory. Program is read into memory and is executed by the kernel. A process is an executing instance of a program.

Process Properties:

• Process identifier / Id
• Command name and arguments
• Env variables
• Current directory
• Owner (user id)
• Tdin, stdout, stderr

Process Id

• Unique identifier of executing process
• When process ends, the id may be reused
• Kernel paging process has ID 0, initial process (init) has ID 1
• Programs can access process id with 
  • int getpid()
• Information about a process is found in the file /proc/<pid>

Commands and Arguments for a Process

• Process has 0 or more arguments for command
• C program starts with arguments passed to a function called main
  • int main(int argc, char **argv)
• argc is the number of arguments, including the command name
• argv is a pointer to an array of pointers to each of the arguments

Process Environment Variables

• Process  inherits its env from its parent
• Env variables are an array of strings A=B
• Some important env variables:
  • PATH=/bin:/usr/bin:
    • Stores list of directories that contains commands to execute
  • USER=<login>
    • Contains the name of the user
  • HOME=/homes/grr
    • Contains the home directory
• Filenames .login or .bashrc can contain env variables to be set with starting shell session
• To set an env variable from shell use:
  • export A=B
    • Globally sets the env var
    • The processes called will get this change
  • A=B
    • Modify local shell environment only
  • Example
    • export PATH=$PATH:/newdir
• env
  • Command that prints out all environment variables

Current Directory

• Every process has current working directory
• File operations open(), fopen() will use working directory to resolve relative path
• Path that starts with “/” is absolute, else relative path
  • /etc/hello.c = absolute
  • Hello.c = relative
• cd / chdir(dir)

Process User Id

• Process always runs on behalf of a user defined by User Id
• UID is inherited from parent
• Only root can change the UID of a process at runtime using:
  • setuid(uid)
• Such a change happens at login time
  • Login program runs as root but after identifying the user and starting a shell, calls OS to change the shell UID to that user
• The su user command changes User Id
• sudo user command runs command with user ID of user
• su = substitute user

 

Execution Modes

• CPU hardware has several possible modes
  • At any one time, in one mode
• Modes specify and/or support:
  • Privilege level
  • Valid instructions
  • Valid memory addresses
  • Size of data items
  • Backwards compatibility

 

Changing execution modes

• Automatic
  • Hardware interrupts
  • OS-specified handlers
• Manual
  • Initiated by software, typically OS
  • System calls, signals, and page faults
  • Sometimes mode can be set by application

 

Modes as rings of protection

Hardware considered as Ring-1 (-1) the highest of privileges; more than OS kernal

• Intel Active Management Technology
• Exists for other architectures as well
• Runs on the Intel Management Engine (ME)
  • Isolated and protected coprocessor
  • Embedded in all current Intel chipsets
  • ARC core
  • Out-of-band access
  • Direct access to Ethernet controller (due to this this has been a target for attacks)
• Requires vPro-enabled CPU/Motherboard/Chipset

 

Ring-1 Vulnerability

• Common vulnerabilities and exposures (CVE)system catalogs issues of concern for computer security
• If Ring-1 can exploited, all software security measures are moot
• If you have control over hardware, then can override all OS/software controls
• Read about CVE-2017-5689

 

 

Unix Shells

What is a shell?

• Shell is a command line interpreter
  • Reads user input and executes commands
  • Makes a computer interactive; dates from mid 1960s
  • Shell is an application program, so it is easy to use an alternative
• Because computer hardware is much cheaper/faster today, there are graphical shells, aka GUIs
  • Gnome, KDE, Windows Explorer, Macintosh Finder
• GUIS can do many of tasks of command line shell, but not all
• Shells are used to make administrative procedures automatic – like backup, program installation, web services 

Shell commands

• When shell has a file to execute a check is made to see if the file:
  • Is known executable format such as a.out or ELF (executable linkable form)
  • Begins with line “#! Interpreter-program” in which case the shell executes the interpreter
• Execute permission must be set for file or interpreter

Some shell programs

• /bin/sh = standard UNIX shell
• /bin/ksh = Korn shell
• /bin/bash = GNU shell
• Bash is widely available in UNIX world, standard in Linux sheell

Conditions

#!/bin/bash
directory="./Scripting"
# if statement to check if directory exists
if [ -d $directory ]; then
    echo "Directory exists"
elif [ -a $directory ]; then
    Echo “File exists, but is not a directory”
else
    echo "Directory does not exist"
fi

Brackets do not touch variables. 

 

Pipes

#!/bin/bash
echo "Retrieving data about" $USER
who | grep $USER

To get list of all environment variables use command printenv

#!/bin/bash
echo $HOME
echo $PATH
echo $USER

 

Arguments

#!/bin/bash
echo $1 $2 $3

The above is using positional parameters. 

 

Parameter(s)	Description
$0	the first positional parameter, equivalent to argv[0] in C, see the first argument
$FUNCNAME	the function name (attention: inside a function, $0 is still the $0 of the shell, not the function name)
$1 … $9	the argument list elements from 1 to 9
${10} … ${N}	the argument list elements beyond 9 (note the parameter expansion syntax!)
$*	all positional parameters except $0, see mass usage
$@	all positional parameters except $0, see mass usage
$#	the number of arguments, not counting $0

File/Directory Checks

if [[ -d $PASSED ]]; then 
echo "$PASSED is a directory" 
elif [[ -f $PASSED ]]; then 
echo "$PASSED is a file" 
else 
echo "$PASSED is not valid" 
exit 1 
fi

Double square brackets is a bash extension to [ ]. It doesn’t require variables to be quoted, not even if they contain spaces.

#!/bin/bash
echo Hello there people! > outfile.txt
cat < outfile.txt

 

File appending

#!/bin/bash
echo "Hello $USER!!" > tmp-message
echo >> tmp-message
echo "Today is" `date` >> tmp-message
echo >> tmp-message
echo "Sincerely," >> tmp-message
echo " Myself" >> tmp-message
/usr/bin/mailx -s "mail-hello" $USER < tmp-message
echo "Message sent."

 

Loops

#!/bin/bash
# for loop
for f in $( ls /var/ ); do
    echo $f
done

COUNT=6
# while loop
while [ $COUNT -gt 0 ]; do
    echo Count: $COUNT
    let COUNT=COUNT-1
done

 

 

Common Unix Commands

 

ssh – secure shell

• ssh [options] [user]@[hostname] [command]
  • Creates secure channel over an unsecure network
  • Primarily for remote login
• Example
  • ssh userloginId@data.cs.purdue.edu

 

man – print manual pages

• man [options] name …
  • Prints manual pages related to the command
• Examples
  • man cp mv
    • Prints cp manual pages, then mv manual pages
  • man -k thread
    • -k = keyword, looks for manual pages with keyword
  • man -s 3 exec
    • Print manual pages of exec from section 3
• Manual pages divided into sections
  • Section 1 = UNIX commands
  • Section 2 = System Calls
  • Section 3 = C language standard library routines
  • .. more Sections
• Example man -s 1 printf and man -s 3 printf
  • printf(1) page for shell print
  • printf(3) page is for the C library printf() function

 

ls -list files

• ls [options] [file …]
  • List directory contents
• Examples
  • ls -al 
    • List all files including hidden
  • ls -R dir
    • Lists recursively all directories and subdirectories in dir

 

mkdir – make directory

• mkdir [options] directory_name
• mkdir -p dir1/dir2/dir3
  • Makes parent directory and subdirectories if not exist

 

cp – copy files

• cp [options] source_file target_file / target_directory
• Examples
  • cp -R dir1 dir2
    • Copy recursively directories and subdirectories

 

mv – move file

• mv [options] source file target_file / target_directory
• Examples
  • mv a.txt b.txt into directory dir1

 

rm – remove directories

• rm [-dfiPRrvW] file ..
  • Attempt to remove
  • User is queried if -w permission and stdout is screen
  • rm removes symbolic links, but does not continue to the field referenced
  • It is an error to try to remove the files “.” and “..”
  • -P means overwrite regular files then remove their directory entries
• Examples
  • rm -f a.txt
    • Attempt to remove and dont print error message
  • rm a.txt b.txt
    • Removes both files
  • rm -r dir1
    • Attempt to remove the file hierarchy rooted at dir1 and all its contents

whereis – locate programs

• Checks the standard binary directories for the specified programs, printing out the paths of any it finds

lore 51 $ whereis apachectl

/p/apache/apachectl   (PATH=/p/apache)

/p/apache-php/bin/apachectl (PATH=/p/apache-php/bin)

/p/apache/man/man8/apachectl.8 (MANPATH=/p/apache/man)

/p/apache-php/man/man8/apachectl.8 (MANPATH=/p/apache-php/man)

 

find – search a directory tree

• Recursively descend the directory hierarchy for each path seeking files that match a Boolean expression

• Find files in the “.” hierarchy with .conf extension:

find . -name “*.conf” -print

• Find files in the . hierarchy and execute the command chmod on found file:

find . -name “*.conf” -exec chmod o+r ‘{}’ \;

 

which – locate a command

• Takes list of filenames and looks for the files that would be executed had these names been given as commands; displays the pathnames of those files. NOte how this is different than whereis.

which ps

/usr/bin/ps

whereis ps

/usr/bin/ps (PATH=/usr/bin)

/usr/ucb/ps (PATH=/usr/ucb)

 

head – display first lines or bytes

• Display first lines or bytes of file/files
• Display the first 10 lines:

head myprog.c

• Display the first 128 bytes:

head -c 128 myprog.c

 

tail – display last lines or bytes

• Display file contents with default of last 10 lines
• Display the last 3 lines:

tail -3 myfile.c

• Display lines of file starting from 15th line

tail +15 myfile.c

• Display last lines of file and continue as lines are appended

tail -f myfile.c

 

grep – global, regular expressions, print

• Search file and print lines that match one or more patterns
• Print the lines in file.txt that contain hello

grep hello file.txt

• Print lines that contain “hello” in any file in the working directory and all subdirectories up to two tree levels deep

grep hello * */* */*/*

 

sed – stream editor

• Reads files (or stdin), modifies files per commands, writes result to stdout
• Command form is:

[address[,address]]function[arguments]

• Example – replace all instances of “current” in file.txt with “new” and redirects the output from stdout to file2.txt

sed s/current/new/g < file.txt > file2.txt

 

awk – pattern scanning and processing language

• Powerful programming language for text processing, typically used for data extraction
• An awk program has a sequence of rules of the form

pattern {action}

…

Default{action}

• Pattern is regular expression
• Action is a sequence of statements that execute when the text matches pattern
• Input lines are viewed as a sequence of fields
• Each input line is matched against each pattern

• Examples
  • Print the first field in each line in file.txt

awk ‘{print $1}’ file.txt

• Print first two fields of each line in reverse order

awk ‘{print $2, $1}’ file.txt

• Add numbers in first column of comma separated values file then print sum and average

awk -F, ‘{s += $1} END { print “Sum”, s,”Average”, s/NR}’ file.csv

Sum is 28, Average is 4

 

I/O redirection – reading

• To redirect input for a program or command,

< file  n< file n is the file descriptor
<< file n<< file n is the file descriptor

• Example

mail jeff@example.edu < my_message

• Be careful using file descriptor numbers greater than 9, as they may conflict with descriptors that the shell uses internally

 

I/O Redirection – writing

• Redirect output from program or command
• >file and n>file redirect output to file
• >>file and n>>file append output to file
• >|file and n>|file override noclobber option if set
• >&n redirects output to file descriptor number n

 

I/O Redirection – pipes

• Pipes enable a series of programs to work together

command_1 | command_2 | … | command_n

• Functions a lot like > except stdout from command_n-1 is redirected to stdin of command_n
• Example

$ls -l | wc -l

• Reports how many lines of text ls output

 

tee command

• tee – copies from stdio to stdout and to any files given as arguments
• any_command | tee save_out
  • Stdout of any_command displayed AND saves a copy of output sent to stdout in save_out
• … | tee save_in | any_command
  • Saves a copy of bytes sent through tee in save_in

 

Command Substitution $( … )

user@host~:$ echo $(seq 1 5)
1 2 3 4 5
# Or, to create 5 new directories:
user@host~:$ mkdir $(seq 1 5)
user@host:~$ echo "42 - 10 is...$(( 42 - 10))"
42 - 10 is...32

 

bc utility

Reads from stdout and evaluates it 

user@host:~$ echo "9.45 / 2.327" | bc
4
user@host:~$ echo "9.45 / 2.327" | bc -l
4.06102277610657498925

IFS

The global variable IFS is what Bash uses to split a string of expanded into separate words…think of it as how Excel knows to split a CSV (comma-separated-values) text file into a spreadsheet: it assumes the commas separate the columns.

user@host:~$ IFS=Z 
user@host:~$ story="The man named Zorro rides a Zebra" 
user@host:~$ echo '>>' $story '<<' 
>> The man named  orro rides a  ebra <<

 

eof