
Several tips on job control on a Linux system: 1. Be "nice": Because all of us are working on the same machine, be considerate while you are running CPU/memory heavy jobs. The rule of thumb is if your job uses more than 25% of the CPU (12 cores) or memory (64G), run it with "nice". All you need to do is add the command "nice " before the command you need to run. It will still run at full speed as long as the resource is available. 2. Know what you are doing: "top" is a convenient way to see what's running. There are many options to control the status page, e.g., you can type "u" on the page to specify users. "ps" is more extensive, and you can see the expanded command that's running. It will display a TON of stuff, so I usually pipe the results into a “grep” to get the commands I want to see. E.g., ps aux | grep python "du" shows the size of folders/files, I usually use "du -sh" for a quick summary. "free" shows the size of used/available memories, Use the flag -g to display the results in GB. You want to pay attention to the used/free memory in the 2nd row (-/+ buffers/cache), because the number in the first row included memory that’s being allocated to cache. 3. killer and renicer: I wrote this two bash commands to either kill a large number of jobs or renice a large number of jobs simultaneously. They are useful when you accidentally launched hundreds of wrong processes. Killer: kill -9 `ps aux | grep USERNAME | grep JOBS | grep -v grep | awk '{print $2}' | tr '\n' ' '` Renicer: sudo renice +20 -p `ps aux | grep USERNAME | grep JOBS | grep -v grep | awk '{print $2}' | tr '\n' ' '` 4. Removing files: In general, be REALLY careful when you use the "rm" command, be REALLY, REALLY careful when you use "rm" in combination with wildcards and -rf flags. Nevertheless, everybody inevitably deleted wrong files at some points of their work, if it's a text file, there might be a chance to recover it. I have attached one method in the attachment; you might want to keep a copy of it. 5. Backup Removing wrong files brings us to the topic of backup. As I mentioned, you will end up losing data at some points of your work (hardware failure, software failure, and more often than not, human errors). So it is important to keep backups of your 1 work. The storage on our Linux server is run on RAID set up and it is backed up by the department every night. Nevertheless, there is no harm to keep a personal copy of backup, especially for your working folders. Each of our desktop computers has a 500 Gb drive so you can back up quite a bit of your work for now. If needed, we can always buy a few external drives for the purpose. I personally use rsync, but it requires manual backup via ssh. If you can find a windows backup program that can back up the server via network, please let me know. The windows version of the rsync can be downloaded from here: http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp 2.
Details
-
File Typepdf
-
Upload Time-
-
Content LanguagesEnglish
-
Upload UserAnonymous/Not logged-in
-
File Pages2 Page
-
File Size-