Skip to content

Profiler for Tensorflow

Jump to bottom
Tianqi Tang edited this page Apr 20, 2018 · 5 revisions

Overview of profiling

Category

There are two major categories of profilers, i.e. instrumenting profilers and sampling profiler.

  • instrumentation: It will modify the program to add performance counters. This is often done using a compiler flag (e.g. gcc -pg, -finstrument-functions), a translation step or can be done at run time (Rational Quantify). Instrumenting profilers can be useful but the code modifications may impact the performance of the code that you are trying to measure. Typically they tend to add a significant overhead on fast functions that are called frequently.

  • sampling: A ``sampling interval", typically in the range of 1ms to 10ms, will pause/interrupt the program and record its state which will typically consist of the backtrace at the current point. By collecting enough of these samples, a representative picture can be achieved on where the program is spending its time.

Tensorflow Profiler Implementation

Usage

For all tfprof views, the profiles are processed with the following procedures

  1. An in-memory data structure is built represent the view.

    • graph view. Graph. This view organizes profile using graph node input/output. Each profiler node corresponds to a TensorFlow graph node.
    • scope view. Tree. This view organizes profile using graph node name scope. Each profiler node corresponds to a TensorFlow graph node.
    • code view. Tree. This view organizes profile using Python call stack. Each profiler node corresponds to all TensorFlow graph nodes created by the profiler node (python code).
    • op view. List. This view organizes profile using operation type. Each profiler node corresponds to all TensorFlow graph nodes belonging to an operation type.

  2. -account_type_regexes is used to first select the nodes that include the specified operation types. An operation has its default type (e.g. MatMul, Conv2D). tfprof also considers the device as operation type. Users can also define customized operation type. Hence, an operation has multiple types. Profiler nodes containing matched types are selected for display and their statistics are aggregated by the parents of the in-memory data structure.

  3. Various -xxx_name_regexes, -min_xxx, -max_depth etc options are then applied to further filter based on profiler node names and statistics. It's no limited operation name. In code view, it's the code string. In op view, it's the operation type name. Different from -account_type_regexes, Statistics are used even if a profiler node is not displayed. For example, in code view, a callee might be hidden, but its statistic is still aggregated by its caller. account_displayed_op_only, however, breaks the rule and only aggregates statistics of displayed names.

  4. Finally, the filtered data structure is output in a format depending on the -output option.

Time

Memory

Tensor memory is usually ref-counted. The memory is released when there is no more reference to it. It will be difficult to track the release of memory. Currently, profiler only tracks the allocation of memory. As a result, the accumulated memory request is usually larger than the peak memory of the overall model. It's recommended to generate timeline to see the allocator memory usage over time.

  • bytes: The memory allocations requested by the operation.
  • peak_bytes: The peak requested memory (not de-allocated) by the operation.
  • residual_bytes: The memory requested by the operation and not de-allocated when Compute finishes.
  • output_bytes: The memory output by the operation. It's not necessarily requested by the current operation. For example, it can be a tensor forwarded from input to output, with in-place mutation.

You can also visualize the memory information through other methods.

  • With op view, it shows you the aggregated output tensor bytes of each operation type.
  • With scope view, you can see the byte of one specific output tensor in the compute graph.
  • With code view, you can see the byte involved in each step of the python call stack.

Reference

Clone this wiki locally