Introduction
- What are we doing
- Before we can tell you how to compile, let's first talk about what we're doing and why.
- It's OK if you don't follow all this -- we're just talking about the why, not the how.
- I don't know your backgrounds. Apologies in advance if you guys already know all this stuff.
- Machine language vs. high-level programming languages
- Machines talk machine language natively.
- "Binaries" (most programs on your system) are in machine language.
- Machine language has three problems
- numeric
- primitive
- Machine language is tightly tied to the system's CPU/processor. Example: x86 (desktop/laptop) machine language is very different from ARM (phone, tablet) machine language.
- Most programming languages are written in a high-level computer language.
- High-level programming languages are relatively expressive, so people can undertand them, while still simple enough for a program to understand.
- High-level languages are usually (largely) machine indendent.
- Computers don't directly understand lanuages other than their machine language. Compilers, interpreters, tokenizers, JREs, and assemblers bridge the gap.
- Software distribution
- Most commercial off-the-shelf (or COTS) software is closed-source. Only binaries are distributed.
- COTS software, especially consumer-grade, usually comes with easy-to-use installers.
- Much free software is open-source. The software is distributed either in source-code form or in binary form with source code available.
- Advantages of open-source
- Transparency -- especially good for scientists
- Reproducibility -- especially good for scientists
- Ability to modify/improve the source code for your requirements
- Reduced vendor lock-in
- Viruses / malware can be seen directly (if you read the source code
- Can support multiple platforms with one distribution
- Can be ported to platforms that the author does not support
- Better interoperability
- Disadvantages of open-source
- Knowledge to build -- but if you're at NCBI, feel free to take advantage of the Unix team. Jira project systems, component Unix.
- Time to build -- ditto.
- Often less polish, and harder to use
- Often no vendor support
- Different modes of programming language operation
- Most programming languages support only a small number of these modes.
- Three times we care about
- Up-front time -- one-time cost to process source code
- Start-up time -- cost every time the program is run to start the program
- Run-time -- how quickly the program runs once started
- Compiled.
- Source code is translated into machine language up-front.
- Compiler -- program (or suite of programs) that translate high-level source code into machine language
- up-front time: big wait to convert to machine language
- start-up time: very fast.
- run-time: often fastest.
- Hard to debug, hard to modify.
- Examples: C, C++, Ada
- Interpreted (pure).
- Source code stays in source code format.
- Interpreter -- program that scans the source code and does what it says.
- up-front time: none.
- start-up time : relatively slow (although each start-up is much faster than the compiler up-front cost)
- run-time: often slower.
- Easy to debug, easy to modify
- "shebang": "#!". Unix convention to locate the interpreter.
- Examples: Perl, Python, Ruby, Bourne shell, Lisp
- Tokenized (variant of interpreted)
- Tokenizers save a step for the interpreter by converting the source code into a format that they have to spend less time upon.
- Source code might be hidden
- up-front time: some, usually not as bad as compiled.
- start-up time: should be faster than pure interpreted, slower than compiled.
- run-time: same as interpreted.
- Examples: Python, Lisp
- Compiled to VM byte code
- Some languages (Java) are designed to run on a "virtual" CPU.
- despite the word "virtual", this is not related to vmware, virtualbox, parallels, or the like
- up-front time: slow, like compiled
- start-up time: slow, like interpreted (need to start the JVM)
- run time: slow, even worse than interpreted
- Examples: Java
- Assembled
- Assembly languages are special languages for people who want to write machine language without learning machine language.
- Still primitive like machine language, but with words instead of numbers.
- Very tightly coupled to machine language
- Assembler: translates to machine lanugage up-front
- Tightly tied to machine architecture (like machine language)
- Up-front time: like compiled
- Start-up time: like compiled
- run time: like compiled (or faster)
- Examples: x86 Assembler, ARM Assembler
- Mixture
- Some languages have multiple options
- C: usually compiled, but interpreters available
- Perl: usually interpreted, but compiler available
- Python: interpreted or tokenized
- Ruby: usually interpreted, but can be compiled to JVM bytecode
- Compiling and building terminology
- Compiler: program that takes source code and converts it into machine language object files
- Often, files have dependencies on other files. They have to be compiled in a certain order.
- Building: more general term for taking a source code project, which may consist of multiple files, and converting them into an installable format.
- There are various tools to help automate builds, so they are done in the correct order, without rebuilding files that don't need to be rebuilt.
- configure tools, if available, configure the package in different ways. Often they will allow you to specify a non-default install location.
- Interpreter -- program that executes high-level source code without first translating into machine code
- General rules
- Read directions!
- Online at the website
- in the distribution file, in a README or INSTALL file
- If you run into an error, look at the first error, not the last error. Often, an initial error causes cascading errors.
- Search google for errors
- Contact the author if needed. Note: they may not reply promptly.
- If you have administrative privileges (root/sudo), you can install into system locations or into your home directory. If you don't have admin privileges, you must install into your homedirectory.
- If you have local support personnel, feel free to contact us! That's why we're here. We're happy to build and install software for you on our central servers.
- Read directions!
- The most important command-line tools for building software
- We will demo these in a bit.
- apt-get: downloads and installs official Ubuntu/Debian software
- dpkg: installs binary software
- wget: downloads files from any URL
- tar: expands most distribution files
- zip: expands zip distribution files
- sudo: runs commands with administrative privileges
- configure (GNU autoconf): configures projects
- Makefile.PL (Perl): configures Perl projects
- make: dependency-aware build tool
- which: tells you where programs are located
- cc, gcc: C compiler
- Setting up a build environment on Ubuntu/Debian
- apt-get update; # update the collection of software available
- apt-get install build-essential; # install lots of build tools
- apt-cache search erlang | less; # search for an erlang package
- apt-get install erlang; # install the erlang package
- Download software
- You probably know how to do this
- Usually via your webbrowser
- Always find the official source for the program. Otherwise, you can end up with out-of-date software.
- Or you can right-click a URL, select "copy link location", and then in a terminal windown: wget PASTED_URL
- Download class notes and samples: wget -m http://frakir.org/~morty/lue/compiling-tutorial/
- Samples
- hello.pl
- shared.tar.gz
- shared.tar.gz
- shared.tgz
- shared.tar.bz2
- shared.zip
- fastqc source from the fastqc fastqc website.
- MOSAIK source from the MOSAIK website.
- cufflinks 2.0.2 source from the
- cufflinks 2.1.1 source from the cufflinks website.
- SRR581069.filt.fastq.gz
- Normal installation steps:
- Check distribution expansion (optional)
- Expand distribution
- Configure
- Actual build
- Test the build
- Install
- Test the installed binaries
- Check distribution expansion (optional)
- Some files expand to a subdirectory. Some expand to the current directory.
- for .tar.gz, .tgz, .tar.Z files: tar tzpf FILE.tar.gz | less
- for .tar.bz2 files: tar tjpf FILE.tar.bz2 | less ; # note "j"
- for .tar files: tar tpf FILE.tar | less
- for zip files: unzip -l FILE.zip | less
- Expand distribution
- If the distribution does not create a subdirectory, make one (mkdir subdir), cd into it (cd subdir), and expand the file using .. before filename below (tar xzpf ../FILE.tar.gz)
- for .tar.gz, .tgz, .tar.Z files: tar xzpf FILE.tar.gz
- for .tar.bz2 files: tar xjpf FILE.tar.bz2 (note the j)
- for .tar files: tar xpf FILE.tar
- for zip files: unzip -q FILE.zip
- Configure
- If a script named "configure" is present, run it: ./configure
- If Makefile.PL is present, run instead: perl Makefile.PL
- If Imakefile is present, run instead: xmkmf -a
- You can pass options to configure and Makefile.PL. The most useful is for non-default installation location.
- ./configure --prefix=$HOME/somepackage
- Actual build
- Normally: make
- Test the build
- Normally: make test
- Install
- If you don't have admin admin priviliges: make install
- If you do have admin priviliges: sudo make install
- Test the installed binaries
- This is program-specific
- Installing binaries
- This talk was mostly about installing source.
- to install binaries in Ubuntu/Debian package format: sudo dpkg -i FILE.deb
- Some common problems (not comprehensive)
- hello.pl: command not found
-
What it means: the system couldn't find your directory.
Why it happens: the command in question isn't in your PATH.
How to fix: either add the directory the command is in to PATH (export PATH=$PATH:/somedir) and add to your .bashrc; or run the command using an explicit path, i.e. ./hello.pl or /somedir/hello.pl
- bash: ./hello.pl: Permission denied
-
What it means: the program or script doesn't have execute permissions
Why it happens: if you do a simple download of a script or binary program file, the download usually loses the "execute" mode setting
How to fix: chmod +x file
- bash: ./hello.pl: /usr/local/bin/perl: bad interpreter: No such file or directory
-
What it means: the "shebang" line of a script points at an interpreter that isn't there.
Why it happens: either the interpreter isn't installed, or it's in a different location.
How to fix: search for the interpreter using apt-cache and install using apt-get, as per above. If the bug continues to occur, the script is probably pointing at the wrong location. Run which perl (or whatever) to find the correct location. For example, /usr/bin/perl. Then edit the script using nano file (or you favorite editor). The first line will say something like #!/usr/local/bin/perl -w". Replace the /usr/local/bin/perl with /usr/bin/perl. Be sure to leave the "#!" in place -- that's a special bit of syntax that needs to be at the beginning of every script.
- error while loading shared libraries: libstdc++.s0.5: cannot open shared object file: No such file or directory
What it means: shared libraries are collections of machine code stored outside binaries. The program depends on a library that isn't where the system is expecting to find it.
Why it happens: systems have a standard search path where they look for libraries. Either the library isn't in the search path (e.g. you installed to your home directory) or the library isn't on the system at all (you installed a binary program that depends on an uninstalled library).
How to fix: find the library in some directory. Set LD_LIBRARY_PATH to include that directory: export LD_LIBRARY_PATH=/somedirectory/path:/another When that is working, add it to your .bashrc. Alternatively, if you have admin privs, you can add the dir to a file in /etc/ld.so.conf.d and run ldconfig. If the library isn't present at all, and you installed from a binary, use apt-cache to search for the library, then apt-cache to install it.
- abundances.h:21:23: fatal error: Eigen/Dense: No such file or directory
What it means: A file (probably a header file) could not be found.
Why it happens: Either the header isn't installed, or the compiler doesn't know where to look.
How to fix: Install the package that contains the header (usually some library followed by -dev, i.e. libboost-dev). If the header is there but in the wrong place, adjust options as needed, edit the Makefile, and/or set CFLAGS.
- inputstreamreader.cpp:66: error: 'memmove' was not declared in this scope
What it means: the cpp file referenced a function -- probably a system function -- that hasn't been declared in an included header file.
Why it happens: sometimes a header that was previously included in one location stops being included in that location. Then you need to add it manually.
How to fix: determine which header file includes the function using a manpage or grep. In this case, string.h. Then manually add #include
near the beginning of the source code file
