Most ubiquitous antivirus products monitor new process from executables in real time and will attempt to terminate their execution if deemed a potential threat. Some of these antivirus products simply do a signature match while some do more sophisticated heuristic or intelligent scanning.
However, there are times when antivirus might turn up a false positive. This is rare, but many software developers must have experienced the slow-down caused merely by the suspending and scanning of the new build from their favorite IDE. Recently my antivirus product let me know he has been very edgy on some Theano python programs with this scan alert.
This rings a bell. I remember the same happened when working with CUDA on Visual Studio. And a test with some sample CUDA programs quickly confirmed my memory.
The solution to get rid of the scan is quite simple. On most antivirus products there is an option to whitelist certain programs from being scanned. And on my Avast installation, simply adding the full file path of the nvcc output will do the trick. Note that doing so may pose certain security risks as this essentially neutralized the protection. So to compensate for the increased risk, the whitelist path should be set as precise as possible, such as including a wildcard filename as shown below.
One last interesting point is, as one of the great benefits from Theano is a high level abstraction of the CUDA layer, it performs some compiling to GPU executable on the CUDA platform using nvcc. Comparing the profiling results obtained before and after antivirus whitelisting shown improvement not only in the overall speed but also in the compile time. With reference to the before-whitelisting profiling result
Function profiling ================== Message: train Time in 10000 calls to Function.__call__: 1.122200e+01s Time in Function.fn.__call__: 1.089400e+01s (97.077%) Time in thunks: 1.069661e+01s (95.318%) Total compile time: 5.477000e+01s Number of Apply nodes: 17 Theano Optimizer time: 3.589900e+01s Theano validate time: 4.000425e-03s Theano Linker time (includes C, CUDA code generation/compiling): 1.772000e+00s Import time 1.741000e+00s
and the whitelisted, optimized results:
Function profiling ================== Message: train Time in 10000 calls to Function.__call__: 9.727999e+00s Time in Function.fn.__call__: 9.469999e+00s (97.348%) Time in thunks: 9.293550e+00s (95.534%) Total compile time: 2.827000e+00s Number of Apply nodes: 17 Theano Optimizer time: 1.935000e+00s Theano validate time: 1.999855e-03s Theano Linker time (includes C, CUDA code generation/compiling): 3.799987e-02s Import time 2.199984e-02s
If Avast only scan the executable before it starts executing, there should be no improvement at all in the compilation. It seems more likely, from analyzing the profiling breakdowns, Avast scans the GPU executable on its creation on the file system by Theano. Turning off Avast’s “File Shield” with no whitelisting triggered no scan alert, therefore confirmed the suspicion.