CudaTools/docs/source/usage.rst

Usage and Examples

This library is broken up into three main parts, as well as a certain compilation and linking framework:

1. Core Examples
2. Array Examples
3. BLAS Examples
4. Compilation and Linking

The Core.h header contains the necessary macros, flags and objects for interfacing with basic kernel launching and the CUDA Runtime API. The Array.h header contains the CudaTools::Array class which provides a device compatible Array-like class with easy memory management. Lastly, the BLAS.h header provides functions BLAS functions through the the cuBLAS library for the GPU, and Eigen for the CPU. Lastly, a templated Makefile is provided which can be used for your own project, after following a few rules.

The usage of this libary will be illustrated through examples, and further details can be found in the other sections. The examples are given in the samples folder. Throughout this documentation, there are a few common terms that may appear. First,we refer to the CPU as the host, and the GPU as the device. So, a host function refers to a function runnable on the CPU, and a device function refers to a function that is runnable on a device. A kernel is a specific function that the host can call to be run on the device.

Core Examples

This file mainly introduces compiler macros and a few classes that are used to improve the syntax between host and device code. To define and call a kernel, there are a few macros provided. For example,

DEFINE_KERNEL(add, int x, int y) {
    printf("Kernel: %i\n", x + y);
}

int main() {
    KERNEL(add, CudaTools::Kernel::basic(1), 1, 1); // Prints 2.
    return 0;
}

The DEFINE_KERNEL(name, ...) macro takes in the function name and its arguments. The second argument in the KERNEL() macro is are the launch parameters for kernel. The launch parameters have several items, but for 'embarassingly parallel' cases, we can simply generate the settings with the number of threads. More detail with creating launch parameters can be found here <CudaTools::Kernel::Settings>. In the above example, there is only one thread. The rest of the arguments are just the kernel arguments. For more detail, see here <Macros>.

Warning

These kernel definitions must be in a file that will be compiled by nvcc. Also, for header files, there is an additional macro DECLARE_KERNEL(name, ...) to declare it and make it available to other files.

Since many applications used classes, a macro is provided to 'convert' a class into being device-compatible. We follow the previous example in a similar fashion.

class intPair {
    DEVICE_CLASS(intPair)
    public:
        int x, y;

        intPair(const int x_, const int y_) : x(x_), y(y_) {
            allocateDevice(); // Allocates memory for this intPair on the device.
            updateDevice().wait(); // Copies the memory on the host to the device and waits until finished.
        };

        ~intPair() { CudaTools::free(that()); };

        HD void swap() {
            int swap = x;
            x = y;
            y = swap;
        };
};

DEFINE_KERNEL(swap, intPair* const pair) { pair->swap(); }

int main() {
    intPair pair(1, 2);
    printf("Before: %u, %u\n", pair.x, pair.y); // Prints 1, 2.

    KERNEL(swap, CudaTools::Kernel::basic(1), pair.that()).wait();
    pair.updateHost().wait(); // Copies the memory from the device back to the host and waits until finished.

    printf("After: %u, %u\n", pair.x, pair.y); // Prints 2, 1.
    return 0;
}

In this example, we create a class called intPair, which is then made available on the device through the DEVICE_CLASS(name) macro. Specifically, that macro introduces a few functions, like allocateDevice(), updateDevice(), updateHost(), and that(). The that() function returns a pointer to the copy on the device. As a result, the programmer must define a destructor that frees the pointer using CudaTools::free(that). For more details, see here <Device Class>.

Warning

The updateDevice() and updateHost() in most cases will need to be explicitly called to push the data on the host to the device, and vice-versa. It is the programmers job to maintain where the 'most recent' copy is. If these are not called, various memory errors can occur. Note that, when passing a pointer to the kernel, it must be the device pointer. Otherwise, an illegal memory access would occur.

The kernel argument list should must consist of pointers to objects, or a non-reference object. Otherwise, compilation will fail. In general this is safer, as it forces the programmer to acknowledge that the device copy is being passed. For the latter case of a non-reference object, you should only do this if there is no issue in creating a copy of the original object. In the above example, we could have done this, but for more complicated classes it may result in unwanted behavior.

Lastly, since the point of classes is usually to have some member functions, to have them available on the device, you must mark them with the compiler macro HD in front.

We also introduce the wait() function, which waits for the command to complete before continuing. Most calls that involve the device are asynchronous, so without proper blocking, operations dependent on a previous command are not guaranteed to run correctly. If the code is compiled for CPU, then everything will run synchronously, as per usual.

Note

Almost all functions that are asynchronous provide an optional 'stream' argument, where you can give the name of the stream you wish to use. Different streams run asynchronous, but operations on the same stream are FIFO. To define a stream to use later, you must call CudaTools::Manager::get()->addStream("myStream") at some point before you use it. For more details, see here <CudaTools::Manager>.

Array Examples

This file introduces the Array class, which is a class that provides automatic memory management between device and host. In particular, it provides functionality on both the host and device while handling proper memory destruction, with many nice features. In particular it supports mimics many features of the Python package NumPy.` We can demonstrate a few here.

DEFINE_KERNEL(times2, const CudaTools::Array<int>& arr) {
    BASIC_LOOP(arr.shape().items()) {
        arr[iThread] *= 2;
    }
}

int main() {
    CudaTools::Array<int> arrRange = CudaTools::Array<int>::range(0, 10);
    CudaTools::Array<int> arrConst = CudaTools::Array<int>::constant(1);
    CudaTools::Array<double> arrLinspace = CudaTools::Array<int>::linspace(0, 5, 10);
    CudaTools::Array<int> arrComma({2, 2}); // 2x2 array.
    arrComma << 1, 2, 3, 4; // Comma initializer if needed.
    std::cout << arrRange << "\n" << arrConst << "\n" << arrLinspace << "\n" << arrComma "\n";

    // Call the kernel multiple times asynchronously. Note: since they share same
    // stream, they are not run in parallel, just queued on the device.
    KERNEL(times2, CudaTools::Kernel::basic(arrRange.shape().items()), arrRange);
    KERNEL(times2, CudaTools::Kernel::basic(arrConst.shape().items()), arrRange);
    KERNEL(times2, CudaTools::Kernel::basic(arrLinspace.shape().items()), arrRange).wait();
    KERNEL(times2, CudaTools::Kernel::basic(arrComma.shape().items()), arrRange).wait();
    arrRange.updateHost();
    arrConst.updateHost();
    arrLinspace.updateHost();
    arrComma.updateHost().wait(); // Only need to wait for the last one, since they have the same stream.

    std::cout << arrRange << "\n" << arrConst << "\n" << arrLinspace << "\n" << arrComma "\n";
    return 0;
}

In this example, we show a few ways to initialize an Array through some static functions. It is templated, so it can (theoretically) support any type. Additionally, you can initialize an empty Array by providing its Shape with an initializer list (ex: {2, 2}). For more details, see here <CudaTools::Array<T>>.

We also note the use of BASIC_LOOP(N), which is a macro for generating the loop automatically on the kernel given the number of threads. It is intended to be used only for "embarassingly parallel" situations and with the CudaTools::Kernel::basic() launch parameters. If compiling for CPU, it will mark the loop with #pragma parallel for and attempt to use OpenMP for parallelism.

The Array also supports other helpful functions, such as multi-dimensional indexing, slicing, and a few other functions.

int main() {
    CudaTools::Array<int> arr = CudaTools::Array<int>::constant(0);
    arr.reshape({4, 5, 5}); // Creates a three dimensional array.

    arr[0][0][0] = 1; // Axis by axis indexing.
    arr[{1, 0, 0}] = 100; // Specific 'coordinate' indexing.
    std::cout << arr << "\n";

    CudaTools::Array<int> arrRange = CudaTools::Array<int>::range(18);
    auto arrSlice = arr.slice({{1, 2}, {1, 4}, {1, 4}}). // Takes a slice of the center.
    std::cout << "Before Copy:\n" << arrSlice << "\n";
    arrSlice = arrRange; // Copies arrRange into arrSlice. (Does NOT replace!)
    std::cout << "After Copy:\n" << arrSlice << "\n";

    std::cout << "Modified: \n" << arr << "\n"; // The original array is modified, since a slice does not copy.

    CudaTools::Array<int> newArr = arr.copy(); // Copies the original Array.
    for (auto it = newArr.begin(); it != newArr.end(); ++it) { // Iterate through the array.
        *it = 1;
    }
    std::cout << "Modified New Array:\n" << newArr << "\n";
    std::cout << "Old Array:\n" << arr << "\n"; // The original array was not modified after a copy.
    return 0;
}

In this example, we demonstrate some of the functionality of the Array. We can do multi-dimensional indexing, take slices of the Array, and iterate through the Array through an iterator, in C++ fashion. Particularly, we need to introduce the concept of a "view" of an Array. An Array either "owns" its data or is a "view" of another Array. You can create a view manually with the .view() function.

Warning

When using the assignment operator, if a view is on the left-hand side, it will perform a copy of the internal data. However, if the Array is an owner, then it will replace the entire Array, and free the old memory. This means any view of that previous array will now point to invalid places in memory. It is responsibility of the programmer to manage this.

BLAS Examples

Compilation and Linking