How to Setup Transparent Squid Proxy Server in Ubuntu

Squid is a fully-featured HTTP/1.0 proxy which is almost (but not quite - we’re getting there!) HTTP/1.1 compliant. Squid offers a rich access control, authorization and logging environment to develop web proxy and content serving applications.

This is a short guide on how to set up a transparent squid proxy server. Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages. Squid has extensive access controls and makes a great server accelerator.

 

Install Squid

Install squid and squid-common

sudo aptitude install squid squid-common

Edit the squid config file.

sudo vi /etc/squid/squid.conf

Set the allowed hosts.

acl internal_network src 192.168.0.0/24 (Where 192.168.0.0/24 is your IP range.)
http_access allow internal_network

Set the correct permissions.

sudo chown -R proxy:proxy /var/log/squid/
sudo chown proxy:proxy /etc/squid/squid.conf


You will need to restart squid for the changes to take affect.

sudo /etc/init.d/squid restart

Now open up your browser and set your proxy to point to your new squid server on port 3128

Authentication

If you wish to use authentication with your proxy you will need to install apache2 utilities

sudo aptitude install squid squid-common apache2-utils

To add your first user you will need to specify -c

sudo htpasswd -c /etc/squid.passwd first_user

Thereafter you add new users with

sudo htpasswd /etc/squid.passwd another_user

Edit the squid config file

sudo vi /etc/squid/squid.conf

Set the the authentication parameters and the acl

auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid.passwd
auth_param basic children 5
auth_param basic realm NFYE Squid proxy-caching web server
auth_param basic credentialsttl 3 hours
auth_param basic casesensitive off


acl users proxy_auth REQUIRED

acl sectionx proxy_auth REQUIRED

http_access allow users

So this is what your squid.conf should look like.

acl all src 0.0.0.0/0.0.0.0
acl internal_network src 192.168.0.0/24
acl users proxy_auth REQUIRED
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563 # https, snews
acl SSL_ports port 873 # rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 631 # cups
acl Safe_ports port 873 # rsync
acl Safe_ports port 901 # SWAT
acl sectionx proxy_auth REQUIRED
acl purge method PURGE
acl CONNECT method CONNECT

http_access allow manager localhost
http_access allow users
http_access allow internal_network
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access deny all
http_reply_access allow all
icp_access allow all

Redirect the all HTTP traffic.

If you would like to redirect the all HTTP traffic through the proxy without needing to set up a proxy manually in all your applications you will need to add some rules

iptables -t nat -A PREROUTING -i eth1 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.0.1:3128
iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128

Where eth1,eth0 are the LAN, WAN devices and 192.168.0.1 is the IP address of your LAN device.

If you wish to monitor the performance of your proxy you can look as some log parser’s (sarg, calamaris, ect.)

SFTP on Ubuntu and Debian in 9 easy steps

In this post, I'll show you how to set up secure ftp (SFTP) access to your Ubuntu server. (Instructions for Debian are very similar: leave out the sudo part and follow these steps as root:)

For this to work, you'll need Ubuntu 8.10 "Intrepid", Debian 5.0 "Lenny" or newer. In this example, mark is the user that can gain superuser rights through sudo. "peter" and a few other users are the ones I want to give sftp access to their personal folder, but not shell access or anything else.

Step 1: If it doesn't exist yet, create a group for the users you want to have sftp access only:
mark@neuskeutel:~$ sudo groupadd sftponly

Step 2: Add user "peter" to this group:
mark@neuskeutel:~$ sudo adduser peter sftponly

Step 3: Install openssh-server if it's not installed yet.
mark@neuskeutel:~$ sudo apt-get install openssh-server

Step 4: Open the default OpenSSH server configuration for editing:
mark@neuskeutel:~$ sudo nano /etc/ssh/sshd_config

Step 5: Change the default sftp server from:
Subsystem sftp /usr/lib/openssh/sftp-server

to
Subsystem sftp internal-sftp

Step 6: Some users can only use sftp, but not other OpenSSH features like remote login. Let's create a rule for that group of users (we'll create the group afterwards). Add the following section to the bottom of /etc/ssh/sshd_config:
Match group sftponly
ChrootDirectory /home/%u
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

Step 7: Pass ownership of peter's directory you want to be sftp accessible to the superuser:
mark@neuskeutel:~$ sudo chown root.root /home/peter

Step 8: Now we change peter's home directory (normally /home/peter) to /:
sudo usermod -d / peter

Step 9: Repeat steps 2, 7 and 8 for any other users that you want to give sftp access.

Did you find this tutorial helpful? Any problems when trying to follow it? Suggestions? Please comment and I'll try to follow up!

What is Linux?

The above definition is probably not going to help you much if you don't know what an Operating system is, and what this UNIX thing is, so let's start at the first major point: What is an Operating system? I promise I won't make it boring!

1.1 What's An Operating System?

Imagine you have a brand new computer. Imagine that nobody had put a disk of any kind into it, ever. That would mean that there was no software installed on the system. If you switched the computer on; It would beep a few times and then tell you that it couldn't start an operating system. The most important software to a computer is one thing: -- the Operating System.

Without an Operating system, not even the world's strongest broadband can connect your computer to the 'net, you couldn't play music, you couldn't write letters. You couldn't do anything!

An operating system is the software that sits between you, the user, and the hardware inside the computer (all the chips etc). If you click the mouse over an icon on your screen, the operating system interprets that you want to load the program that you are clicking on. For all of this to happen, The Operating system (some times referred to as the OS or O/S) must know how to use a screen (to show you what's going on), to use a mouse (so you can move it around and click with it), to use your hard disk drive (to load up the data from it). It must also need to know pretty much everything else about the hardware installed inside your PC, ie: RAM, Floppy/CD drives, keyboards, joysticks, sound cards, printers, scanners, etc.

So when you start typing a letter, for example, you have already loaded up a word processing piece of software. This software is called application software and is running 'on top' of the Operating System, but nonetheless, all of the time whilst the word processor application is running, it talks constantly to the O/S for vital information.

Okay, we've pretty much established that an O/S is vital, but what else does an O/S do: Probably the most basic and yet essential tasks of an operating system is the job of managing our files and data. A basic O/S should be able to do the following with files and directories:

• Create them


• Move them to other directories


• Rename them


• Copy them


• Delete/remove them


• Send and receive files to/from other devices such as Printers/Scanners


• ...and a bit more.

Now you have the idea of what an Operating system is, let's find out about a specific type of operating system called UNIX...

1.2 What is UNIX? (And a little bit of history)


The operating system UNIX began life in 1969, in Bell Labs, a division of the American Telephone firm, AT&T. There are now many different types of UNIX, making it one of the longest running commercial operating systems available, way longer than Microsoft Windows or Apple MacOS.

• Unix History Timeline

Linux is just one type of UNIX which is most famously known for being a free (as in free speech) derivative of UNIX. Most of UNIX's different flavours are still being updated and are still in use all over the world today. Here are just some popular manufacturers and brands of UNIX, that you may or may not have heard of before:


Sun Microsystems: Solaris


Hewlett Packard: HP/UX

Berkley University: NetBSD and FreeBSD

With the exception of FreeBSD, there was (and still is) a pretty grand fee to own one of the above versions of UNIX. Mainly large commercial organisations and universities have traditionally used these UNIX variants, however Linux appears to be replacing traditional UNIX on a lot of corporate systems due to it's proven track record, it's growing reputation as a contender to UNIX, and it's low price tag, which can often be free.
UNIX is good because it is a true multi-tasking, multi-user operating system. This means that it can do more than one thing at a time and it can provide all it's services to lots of users at the same time. Modern day workplaces rely on servers to provide a central resource of information and connectivity to users.

So, Unix is pretty clever, huh?

Well, yes. It is, but Unix was also traditionally a pretty boring system that involved learning lots of commands.
Why don't we all use UNIX today if it's so good?


In 1981, a small company based in Seattle called Microsoft released an operating system, which through chance (Digital Research were supposed to get the contract), were taken on by IBM to go on their new home/small office based computer: the IBM PC (or Personal Computer). This system was also not graphical. It required commands, in a similar format to UNIX or CP/M, but they were a little easier to use, at the cost of being simpler and less powerful. The main pitfall of MS-DOS - (Microsoft's PC Operating System) was, that it had no multi-user, multi-tasking or networking support as standard. By the early 1990's, this was really starting to wear on PC users. UNIX still had far more power than most operating systems of the time, it was just way too expensive, and legal issues between UNIX vendors licensing UNIX was causing headaches and therefore did not have much exposure outside of large organisations, educational establishments and government offices.



During the 80's, a firm called Apple had released another computer, which was seperate from the PC, and did not run any PC software, because it relied on it's own O/S, named MacOS. This time, Apple had decided to make an operating system that was graphical, and later, incorporated colour, pictures, icons and even sounds! Instead of typing everything into the keyboard as commands, the same actions could be made as clicks and movements with a mouse. A device that the PC had, but rarely used.
In a similar sense, and time, the UNIX world, still very different to the market of the PC and the Mac, had a graphical front-end to it's command-line console, it was called X, or 'The X Window System'.

In 1990, Microsoft eventually released Windows 3.0 (versions 1 and 2 did not sell well) Windows at the time was a 16-bit, single-tasking, single user, graphical interface built on-top of MS-DOS. but UNIX still prevailed: it was multi-user, multi-tasking and it worked on 32 or 64-bit platforms.It took until 1995, with the advent of Microsoft Windows 95 for Windows to finally go 32 bit, multi tasking, and capable of being multi-user (although not best suited. Windows NT came along shortly after, to do that job).

1.3 How does Linux differ from UNIX?

During the time from 1991 to 1995, many computing or engineering students were accustomed to the power of UNIX and X, at university. In Uni, most students had wonderful new things like E-Mail, The Internet and more. At home, they would have to make do with their 16 bit computers, waiting for all these powers to come to their homes one day.

Enter: Linus Torvalds

Linus was, in 1991, a student in Finland studying Computer programming at The University of Helsinki. Linus used UNIX at University on a daily basis. He got bored of his 386 PC running MS-DOS, and decided to start his own kernel, which is the name for the code at the heart of every operating system that talks to the hardware directly. He wanted to distribute the software freely, because it was a hobby, not a commercial product, and also to see what others thought of it. He finished the first Linux kernel in late 1991. Not only had he made a 32 bit kernel, in which programs could be run he made it do quite a lot to make it look and feel like Unix, but he didn't have any software to run on it.

 

• Read Linus' very first post to the Internet in 1991, about the creation of what would become Linux

Luckily, an ex-student in the USA, by the name of Richard Stallman had created a team of programmers devoted to free software, he called this the Free Software Foundation, who believed in making software free to distribute, and free to obtain the source code along with it so that others could make improvements to the software through the Internet. The GNU GPL (General Public License) that the Free Software Foundation made, also stated that the authors of the software could charge for the software, as long as they are willing for it to be freely distributed. By creating community-based software, that has open standards and is subject to peer review, the quality of the software would be good. The opportunities for profit would come from different avenues such as support and consultancies.
Stallman had been busy making a whole suite of software, for example: an editor called emacs, which is very popular today, and the bash (Bourne Again Shell), a command line interface based upon the original Bourne Shell, that comes with the BSD variant of UNIX. The FSF's software was entirely based upon the UNIX software suite, and essentially improved on it. In 1991, the only thing that the FSF were missing to make it a fully fledged operating system was The Kernel.

Linus altered the code to work on his platform so that the FSF's code would work with his new kernel, that he ended up calling Linux.

Thunderbird localmail Spool

I was in a bit of a bind the other day when I learned that an IMAP server I was using was going to dissapear. I wanted to backup all of my mail, but had too many messages hosted on the IMAP server to copy from one to another, so I decided I’d download them all locally first and then deal with them later. I used fetchmail to download all of my messages from the IMAP main folder to my local spool, and copied over all of the sub folders because I was in a rush and needed to copy them quickly.

Once they were in my local mail spool, I wanted to get them into Thunderbird, but learned that the option I had once used to read my localmail had gone. There used to be an option for it in the GUI, but somewhere along the line it got removed. After a bit of Googling, I found: http://lists.freebsd.org/pipermail/freebsd-questions/2003-December/027652.html

This worked for me, and should work for any one that wants to use Thunderbird to read local mail.

Basically, you add a new mail account in Thunderbird as usual, then close it. Get into your local profile directory by using

cd .thunderbird/[tab]

then

vi prefs.js

find your new mail server, with the bogus name and change the hostname to localhost, change your name to <yourusername>@localhost, the server type to movemail, and change the userName to your username. It should look something like the following:

user_pref(“mail.server.server4.hostname”, “localhost”);
user_pref(“mail.server.server4.name”, “owen@localhost”);
user_pref(“mail.server.server4.type”, “movemail”);
user_pref(“mail.server.server4.userName”, “owen”);

Once thats done, you can restart Thunderbird and fetch your mail as usual. From there you can do as you wish with your messages.

Excellent! My question really is why was the GUI option removed from Thunderbird? Whatever the answer this method still works, so if you need to, use it while you still can!

Top 5 Email Client For Linux, Mac OS X, and Windows Users

Linux comes with various GUI based email client to stay in touch with your friends and family, and share information in newsgroups with other users. The following software is similar to Outlook Express or Windows Live Mail and is used by both home and office user.

Webmail interfaces allow users to access their mail with any standard web browser, from any computer, rather than relying on an e-mail client.

However, e-mail client remains extremely popular in a large corporate environment, small business, home and power users. An e-mail client (also mail user agent (MUA)) is a frontend computer program used to manage e-mail. Mail can be stored on the client, on the server side, or in both places. Standard formats for mailboxes include Maildir and mbox.

The following are top five amazing piece of cross-platform software from various projects to make your life easy with wide variety of plug-ins / add-ons.

#1: Mozilla Thunderbird

It is an e-mail and news cross-platform client software package by Mozilla Foundation. Thunderbird can manage multiple e-mail, newsgroup and RSS accounts and supports multiple identities within accounts. Features like quick search, saved search folders , advanced message filtering, message grouping, and labels help manage and find messages. Just like Firefox, the tons of extensions and themes for this client makes it very secure and flexible to to enhance your productivity.

Fig.01: Mozilla Thunderbird

=> Download Mozilla Thunderbird

#2: Claws Mail

Claws Mail is a free, GTK+-based, open source email and news client. It is very light lightweight. Like Firefox , the wide variety of plug-ins for this email client makes it very flexible and secure. Claws Mail runs on Windows, Mac OS X and Unix-like systems such as Linux, BSD, and Solaris.

Fig.02: Claws Mail in Action

=> Download Claws Mail

#3: Spicebird

Update: Ed - 23/June/2010 ~ Spicebird client is no longer supported on Mac OS X. We will replace this email client with another supported product shortly.

Spicebird is a collaboration client that provides integrated access to email, contacts, calendaring and instant messaging in a single application. It provides easy access to various web services while retaining all the advantages of a desktop application. It is developed by an Indian company called Synovel. It is a free, open source and cross-platform software.

Fig.03: Spicebird in Action (image credit Spicebird project)

=> Download Spicebird

#4: Zimbra Collaboration Suite (Open Source Version)

Zimbra is a client and server platform for messaging and collaboration. The web client integrates email, contacts, shared calendar, VoIP, and online document authoring in a rich browser-based interface. This is more like MS-Exchange and Outlook combo. In other words it is compatible with proprietary clients such as Microsoft Outlook and Apple Mail, both through proprietary connectors, as well as the open-source Novell Evolution, so that mail, contacts, and calendar items can be synchronised from these to the ZCS server. Zimbra also provides native two-way sync to many mobile devices such as Nokia Eseries, BlackBerry, Windows Mobile, iPhone with 2.0 software.

Fig.04: Zimbra (credit offical Zimbra website)

=> Download Zimbra Collaboration Suite (Open Source Version)

#5: Sylpheed

Sylpheed is a free, GTK+-based, open source email and news client. It is very light lightweight. Sylpheed runs on Windows, Mac OS X and Unix-like systems such as Linux, and BSD.

Fig.05: Sylpheed in Action

=> Download sylpheed

Comparison of E-mail Clients - Essential Features

The following tables compare general and technical information between e-mail client programs.

Feature	Thunderbird	Claws Mail	Spicebird	Zimbra	Sylpheed
Cross-platform	Y	Y	Y	Y	Y
License	MPL, MPL/GNU GPL/GNU LGPL	GPL	MPL, MPL/GNU GPL/GNU LGPL	MPL (server) and ZPL (client)	GPL/LGPL
Cost	Free	Free	Free	Free	Free
Authentication	Y	Y	Y	Y	Y
SSL and TLS	Y	Y	Y	Y	Y
Image blocking	Y	Y	Y	Y	Y
Junk filtering	Y	Y	Y	Y	Y
Phishing filtering	Y	Y	Y	Y	?
Add-ons	Y	Y	Y	?	N
Thread view	Y	Y	Y	Y	Y
PGP support	Y	Y	Y	?	Y
Label Messages	Y	Y	?	Y	Y
Spell Checking	Y	Y	?	Y	N
Signatures	Y	Y	Y	Y	Y
Scheduled message	Y	?	?	?	?
Message templates	Y	Y	?	Y	Y
Database	mbox	MH, mbox	?	File system	MH
POP3	Y	Y	Y	Y	Y
IMAP4	Y	Y	Y	Y	Y
SMTP	Y	Y	Y	Y	Y
NNTP (News)	Y	Y	Y	Y	Y
RSS Feed	Y	Y	Y	Y	N
LDAP	Y	Y	Y	Y	N
iCalendar	Y	?	Y	Y	N
Paid Support	?	?	?	?	Y

Y = supported; N = not supported; ? = unknown; Privacy feature; Security features; Productivity features; Cross-platform - runs on Mac OS X, Windows and UNIX like operating systems.

Other Email Clients For UNIX Like Operating Systems

1. SeaMonkey - Mozilla SeaMonkey is an all-in-one Internet application suite that includes an Internet browser, email and newsgroup client, HTML editor, IRC chat, and web development tools. It includes a pop-up blocker, junk mail controls, and a tabbed interface.


2. Pine (Alpine) - Alpine is a rewrite of the Pine Message System that adds support for Unicode and other features. Alpine is meant to be suitable for both inexperienced email users and the most demanding of power users.


3. Evolution or Novell Evolution - Evolution provides integrated mail, addressbook and calendaring functionality to users of the GNOME desktop.

Our Recommendations:

1. Claws Mail - Highly recommended for netbook user due to lightweight usage.


2. Mozilla Thunderbird - The wide variety of add-on for this email client makes it very flexible, secure and easy to use. Highly recommended for desktop and power users.


3. Zimbra Collaboration Suite ~ Open Source Edition or Businesses Editon - Highly recommended for business and corporate users due to its support for a broad range of email clients and mobile devices via "over the air" sync.

All of the e-mail client listed above used by me at one point or another. If you know of, or use, another e-mail client that offers better features than those mentioned here, tell us in the comments.

20 Linux System Monitoring Tools Every SysAdmin Should Know

Need to monitor Linux server performance? Try these built-in command and a few add-on tools. Most Linux distributions are equipped with tons of monitoring. These tools provide metrics which can be used to get information about system activities. You can use these tools to find the possible causes of a performance problem. The commands discussed below are some of the most basic commands when it comes to system analysis and debugging server issues such as:

1. Finding out bottlenecks.


2. Disk (storage) bottlenecks.


3. CPU and memory bottlenecks.


4. Network bottlenecks.

#1: top - Process Activity Command

The top program provides a dynamic real-time view of a running system i.e. actual process activity. By default, it displays the most CPU-intensive tasks running on the server and updates the list every five seconds.

Fig.01: Linux top command

Commonly Used Hot Keys

The top command provides several useful hot keys:

Hot Key	Usage
t	Displays summary information off and on.
m	Displays memory information off and on.
A	Sorts the display by top consumers of various system resources. Useful for quick identification of performance-hungry tasks on a system.
f	Enters an interactive configuration screen for top. Helpful for setting up top for a specific task.
o	Enables you to interactively select the ordering within top.
r	Issues renice command.
k	Issues kill command.
z	Turn on or off color/mono

=> Related: How do I Find Out Linux CPU Utilization?

#2: vmstat - System Activity, Hardware and System Information

The command vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity.
# vmstat 3
Sample Outputs:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 2540988 522188 5130400    0    0     2    32    4    2  4  1 96  0  0
 1  0      0 2540988 522188 5130400    0    0     0   720 1199  665  1  0 99  0  0
 0  0      0 2540956 522188 5130400    0    0     0     0 1151 1569  4  1 95  0  0
 0  0      0 2540956 522188 5130500    0    0     0     6 1117  439  1  0 99  0  0
 0  0      0 2540940 522188 5130512    0    0     0   536 1189  932  1  0 98  0  0
 0  0      0 2538444 522188 5130588    0    0     0     0 1187 1417  4  1 96  0  0
 0  0      0 2490060 522188 5130640    0    0     0    18 1253 1123  5  1 94  0  0

Display Memory Utilization Slabinfo

# vmstat -m

Get Information About Active / Inactive Memory Pages

# vmstat -a
=> Related: How do I find out Linux Resource utilization to detect system bottlenecks?

#3: w - Find Out Who Is Logged on And What They Are Doing

w command displays information about the users currently on the machine, and their processes.
# w username
# w vivek
Sample Outputs:

 17:58:47 up 5 days, 20:28,  2 users,  load average: 0.36, 0.26, 0.24
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    10.1.3.145       14:55    5.00s  0.04s  0.02s vim /etc/resolv.conf
root     pts/1    10.1.3.145       17:43    0.00s  0.03s  0.00s w

#4: uptime - Tell How Long The System Has Been Running

The uptime command can be used to see how long the server has been running. The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.
# uptime
Output:

 18:02:41 up 41 days, 23:42,  1 user,  load average: 0.00, 0.00, 0.00

1 can be considered as optimal load value. The load can change from system to system. For a single CPU system 1 - 3 and SMP systems 6-10 load value might be acceptable.

#5: ps - Displays The Processes

ps command will report a snapshot of the current processes. To select all processes use the -A or -e option:
# ps -A
Sample Outputs:

  PID TTY          TIME CMD
    1 ?        00:00:02 init
    2 ?        00:00:02 migration/0
    3 ?        00:00:01 ksoftirqd/0
    4 ?        00:00:00 watchdog/0
    5 ?        00:00:00 migration/1
    6 ?        00:00:15 ksoftirqd/1
....
.....
 4881 ?        00:53:28 java
 4885 tty1     00:00:00 mingetty
 4886 tty2     00:00:00 mingetty
 4887 tty3     00:00:00 mingetty
 4888 tty4     00:00:00 mingetty
 4891 tty5     00:00:00 mingetty
 4892 tty6     00:00:00 mingetty
 4893 ttyS1    00:00:00 agetty
12853 ?        00:00:00 cifsoplockd
12854 ?        00:00:00 cifsdnotifyd
14231 ?        00:10:34 lighttpd
14232 ?        00:00:00 php-cgi
54981 pts/0    00:00:00 vim
55465 ?        00:00:00 php-cgi
55546 ?        00:00:00 bind9-snmp-stat
55704 pts/1    00:00:00 ps

ps is just like top but provides more information.

Show Long Format Output

# ps -Al
To turn on extra full mode (it will show command line arguments passed to process):
# ps -AlF

To See Threads ( LWP and NLWP)

# ps -AlFH

To See Threads After Processes

# ps -AlLm

Print All Process On The Server

# ps ax
# ps axu

Print A Process Tree

# ps -ejH
# ps axjf
# pstree

Print Security Information

# ps -eo euser,ruser,suser,fuser,f,comm,label
# ps axZ
# ps -eM

See Every Process Running As User Vivek

# ps -U vivek -u vivek u

Set Output In a User-Defined Format

# ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
# ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
# ps -eopid,tt,user,fname,tmout,f,wchan

Display Only The Process IDs of Lighttpd

# ps -C lighttpd -o pid=
OR
# pgrep lighttpd
OR
# pgrep -u vivek php-cgi

Display The Name of PID 55977

# ps -p 55977 -o comm=

Find Out The Top 10 Memory Consuming Process

# ps -auxf | sort -nr -k 4 | head -10

Find Out top 10 CPU Consuming Process

# ps -auxf | sort -nr -k 3 | head -10

#6: free - Memory Usage

The command free displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel.
# free
Sample Output:

            total       used       free     shared    buffers     cached
Mem:      12302896    9739664    2563232          0     523124    5154740
-/+ buffers/cache:    4061800    8241096
Swap:      1052248          0    1052248

=> Related: :

1. Linux Find Out Virtual Memory PAGESIZE


2. Linux Limit CPU Usage Per Process


3. How much RAM does my Ubuntu / Fedora Linux desktop PC have?

#7: iostat - Average CPU Load, Disk Activity

The command iostat report Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems (NFS).
# iostat
Sample Outputs:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 	06/26/2009
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.50    0.09    0.51    0.03    0.00   95.86
Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              22.04        31.88       512.03   16193351  260102868
sda1              0.00         0.00         0.00       2166        180
sda2             22.04        31.87       512.03   16189010  260102688
sda3              0.00         0.00         0.00       1615          0

=> Related: : Linux Track NFS Directory / Disk I/O Stats

#8: sar - Collect and Report System Activity

The sar command is used to collect, report, and save system activity information. To see network counter, enter:
# sar -n DEV | more
To display the network counters from the 24th:
# sar -n DEV -f /var/log/sa/sa24 | more
You can also display real time usage using sar:
# sar 4 5
Sample Outputs:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 		06/26/2009
06:45:12 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
06:45:16 PM       all      2.00      0.00      0.22      0.00      0.00     97.78
06:45:20 PM       all      2.07      0.00      0.38      0.03      0.00     97.52
06:45:24 PM       all      0.94      0.00      0.28      0.00      0.00     98.78
06:45:28 PM       all      1.56      0.00      0.22      0.00      0.00     98.22
06:45:32 PM       all      3.53      0.00      0.25      0.03      0.00     96.19
Average:          all      2.02      0.00      0.27      0.01      0.00     97.70

=> Related: : How to collect Linux system utilization data into a file

#9: mpstat - Multiprocessor Usage

The mpstat command displays activities for each available processor, processor 0 being the first one. mpstat -P ALL to display average CPU utilization per processor:
# mpstat -P ALL
Sample Output:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in)	 	06/26/2009
06:48:11 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
06:48:11 PM  all    3.50    0.09    0.34    0.03    0.01    0.17    0.00   95.86   1218.04
06:48:11 PM    0    3.44    0.08    0.31    0.02    0.00    0.12    0.00   96.04   1000.31
06:48:11 PM    1    3.10    0.08    0.32    0.09    0.02    0.11    0.00   96.28     34.93
06:48:11 PM    2    4.16    0.11    0.36    0.02    0.00    0.11    0.00   95.25      0.00
06:48:11 PM    3    3.77    0.11    0.38    0.03    0.01    0.24    0.00   95.46     44.80
06:48:11 PM    4    2.96    0.07    0.29    0.04    0.02    0.10    0.00   96.52     25.91
06:48:11 PM    5    3.26    0.08    0.28    0.03    0.01    0.10    0.00   96.23     14.98
06:48:11 PM    6    4.00    0.10    0.34    0.01    0.00    0.13    0.00   95.42      3.75
06:48:11 PM    7    3.30    0.11    0.39    0.03    0.01    0.46    0.00   95.69     76.89

=> Related: : Linux display each multiple SMP CPU processors utilization individually.

#10: pmap - Process Memory Usage

The command pmap report memory map of a process. Use this command to find out causes of memory bottlenecks.
# pmap -d PID
To display process memory information for pid # 47394, enter:
# pmap -d 47394
Sample Outputs:

47394:   /usr/bin/php-cgi
Address           Kbytes Mode  Offset           Device    Mapping
0000000000400000    2584 r-x-- 0000000000000000 008:00002 php-cgi
0000000000886000     140 rw--- 0000000000286000 008:00002 php-cgi
00000000008a9000      52 rw--- 00000000008a9000 000:00000   [ anon ]
0000000000aa8000      76 rw--- 00000000002a8000 008:00002 php-cgi
000000000f678000    1980 rw--- 000000000f678000 000:00000   [ anon ]
000000314a600000     112 r-x-- 0000000000000000 008:00002 ld-2.5.so
000000314a81b000       4 r---- 000000000001b000 008:00002 ld-2.5.so
000000314a81c000       4 rw--- 000000000001c000 008:00002 ld-2.5.so
000000314aa00000    1328 r-x-- 0000000000000000 008:00002 libc-2.5.so
000000314ab4c000    2048 ----- 000000000014c000 008:00002 libc-2.5.so
.....
......
..
00002af8d48fd000       4 rw--- 0000000000006000 008:00002 xsl.so
00002af8d490c000      40 r-x-- 0000000000000000 008:00002 libnss_files-2.5.so
00002af8d4916000    2044 ----- 000000000000a000 008:00002 libnss_files-2.5.so
00002af8d4b15000       4 r---- 0000000000009000 008:00002 libnss_files-2.5.so
00002af8d4b16000       4 rw--- 000000000000a000 008:00002 libnss_files-2.5.so
00002af8d4b17000  768000 rw-s- 0000000000000000 000:00009 zero (deleted)
00007fffc95fe000      84 rw--- 00007ffffffea000 000:00000   [ stack ]
ffffffffff600000    8192 ----- 0000000000000000 000:00000   [ anon ]
mapped: 933712K    writeable/private: 4304K    shared: 768000K

The last line is very important:

• mapped: 933712K total amount of memory mapped to files


• writeable/private: 4304K the amount of private address space


• shared: 768000K the amount of address space this process is sharing with others

=> Related: : Linux find the memory used by a program / process using pmap command

#11 and #12: netstat and ss - Network Statistics

The command netstat displays network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. ss command is used to dump socket statistics. It allows showing information similar to netstat. See the following resources about ss and netstat commands:

• ss: Display Linux TCP / UDP Network and Socket Information


• Get Detailed Information About Particular IP address Connections Using netstat Command

#13: iptraf - Real-time Network Statistics

The iptraf command is interactive colorful IP LAN monitor. It is an ncurses-based IP LAN monitor that generates various network statistics including TCP info, UDP counts, ICMP and OSPF information, Ethernet load info, node stats, IP checksum errors, and others. It can provide the following info in easy to read format:

• Network traffic statistics by TCP connection


• IP traffic statistics by network interface


• Network traffic statistics by protocol


• Network traffic statistics by TCP/UDP port and by packet size


• Network traffic statistics by Layer2 address

Fig.02: General interface statistics: IP traffic statistics by network interface

Fig.03 Network traffic statistics by TCP connection

#14: tcpdump - Detailed Network Traffic Analysis

The tcpdump is simple command that dump traffic on a network. However, you need good understanding of TCP/IP protocol to utilize this tool. For.e.g to display traffic info about DNS, enter:
# tcpdump -i eth1 'udp port 53'
To display all IPv4 HTTP packets to and from port 80, i.e. print only packets that contain data, not, for example, SYN and FIN packets and ACK-only packets, enter:
# tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
To display all FTP session to 202.54.1.5, enter:
# tcpdump -i eth1 'dst 202.54.1.5 and (port 21 or 20'
To display all HTTP session to 192.168.1.5:
# tcpdump -ni eth0 'dst 192.168.1.5 and tcp and port http'
Use wireshark to view detailed information about files, enter:
# tcpdump -n -i eth1 -s 0 -w output.txt src or dst port 80

#15: strace - System Calls

Trace system calls and signals. This is useful for debugging webserver and other server problems. See how to use to trace the process and see What it is doing.

#16: /Proc file system - Various Kernel Statistics

/proc file system provides detailed information about various hardware devices and other Linux kernel information. See Linux kernel /proc documentations for further details. Common /proc examples:
# cat /proc/cpuinfo
# cat /proc/meminfo
# cat /proc/zoneinfo
# cat /proc/mounts

17#: Nagios - Server And Network Monitoring

Nagios is a popular open source computer system and network monitoring application software. You can easily monitor all your hosts, network equipment and services. It can send alert when things go wrong and again when they get better. FAN is "Fully Automated Nagios". FAN goals are to provide a Nagios installation including most tools provided by the Nagios Community. FAN provides a CDRom image in the standard ISO format, making it easy to easilly install a Nagios server. Added to this, a wide bunch of tools are including to the distribution, in order to improve the user experience around Nagios.

18#: Cacti - Web-based Monitoring Tool

Cacti is a complete network graphing solution designed to harness the power of RRDTool's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices. It can provide data about network, CPU, memory, logged in users, Apache, DNS servers and much more. See how to install and configure Cacti network graphing tool under CentOS / RHEL.

#19: KDE System Guard - Real-time Systems Reporting and Graphing

KSysguard is a network enabled task and system monitor application for KDE desktop. This tool can be run over ssh session. It provides lots of features such as a client/server architecture that enables monitoring of local and remote hosts. The graphical front end uses so-called sensors to retrieve the information it displays. A sensor can return simple values or more complex information like tables. For each type of information, one or more displays are provided. Displays are organized in worksheets that can be saved and loaded independently from each other. So, KSysguard is not only a simple task manager but also a very powerful tool to control large server farms.

Fig.05 KDE System Guard {Image credit: Wikipedia}

See the KSysguard handbook for detailed usage.

#20: Gnome System Monitor - Real-time Systems Reporting and Graphing

The System Monitor application enables you to display basic system information and monitor system processes, usage of system resources, and file systems. You can also use System Monitor to modify the behavior of your system. Although not as powerful as the KDE System Guard, it provides the basic information which may be useful for new users:

• Displays various basic information about the computer's hardware and software.


• Linux Kernel version


• GNOME version


• Hardware


• Installed memory


• Processors and speeds


• System Status


• Currently available disk space


• Processes


• Memory and swap space


• Network usage


• File Systems


• Lists all mounted filesystems along with basic information about each.

Fig.06 The Gnome System Monitor application

Bonus: Additional Tools

A few more tools:

• nmap - scan your server for open ports.


• lsof - list open files, network connections and much more.


• ntop web based tool - ntop is the best tool to see network usage in a way similar to what top command does for processes i.e. it is network traffic monitoring software. You can see network status, protocol wise distribution of traffic for UDP, TCP, DNS, HTTP and other protocols.


• Conky - Another good monitoring tool for the X Window System. It is highly configurable and is able to monitor many system variables including the status of the CPU, memory, swap space, disk storage, temperatures, processes, network interfaces, battery power, system messages, e-mail inboxes etc.


• GKrellM - It can be used to monitor the status of CPUs, main memory, hard disks, network interfaces, local and remote mailboxes, and many other things.


• vnstat - vnStat is a console-based network traffic monitor. It keeps a log of hourly, daily and monthly network traffic for the selected interface(s).


• htop - htop is an enhanced version of top, the interactive process viewer, which can display the list of processes in a tree form.


• mtr - mtr combines the functionality of the traceroute and ping programs in a single network diagnostic tool.