Non-determinism Docs (#2732) by jkschin · Pull Request #10636 · tensorflow/tensorflow

jkschin · 2017-06-11T17:27:22Z

Fixes #2732. Wrote a short tutorial on non-determinism in TensorFlow due to GPU reductions.

Wrote a short tutorial on non-determinism in TensorFlow due to GPU reductions.

tensorflow-jenkins · 2017-06-11T17:27:23Z

Can one of the admins verify this patch?

michaelisard · 2017-06-14T21:46:57Z

This looks fine to me, but I will pass to @zheng-xq in case there is a different best practice to ensure determinism when it is required.

zheng-xq · 2017-06-15T01:04:56Z

Erich is working on a design to enable better deterministic support. Adding him to make sure the language is consistent with the direction we are taking.

jkschin · 2017-06-21T15:50:20Z

@ekelsen any updates?

ekelsen · 2017-06-21T16:21:44Z

Reductions will soon be deterministic on the GPU without performance loss. For some types (float16, complex64, complex128, bool) the performance will increase by 30-1000x.

ekelsen · 2017-06-21T16:24:33Z

This is an awesome doc and thank you for taking the time to write it. The non-deterministic nature of the GPU reductions has been confusing people for a long time.

However, I'm not sure that it makes sense to pull this in for the brief period of time before the new reductions go in.

jkschin · 2017-06-22T01:02:10Z

Sounds awesome, deterministic reductions with no performance loss. If determinism is guaranteed from then on, then perhaps it doesn't make sense to pull this in for the brief period.

It might make sense to still pull this in and add more explanation on how your side has made it deterministic with no performance loss, for those who intend to understand it.

Feel free to close this PR if it's the best option at this point.

drpngx · 2017-06-26T21:46:48Z

@ekelsen I am closing this PR since it's going to be obsolete pretty soon. Feel free to re-open if you think it's worth having in the meantime.
@tfboyd just FYI
@jkschin nice doc! I wish we had earlier :-)

jkschin · 2017-06-27T02:18:48Z

@drpngx cheers! Shall look for the next PR to do :)

alexklibisz · 2017-07-14T14:23:46Z

@ekelsen Hi, do you maybe have an update for when the deterministic functionality will be available? Is it something implemented in TF itself or a feature of CUDNN? Thank you

ekelsen · 2017-09-13T21:11:57Z

Reductions are deterministic now. l2loss is deterministic. softmax will be shortly. cross entropy sometime soon.

ekelsen · 2017-09-13T21:13:05Z

The backward pass of the convolutions is not deterministic, but this basically NVIDIA's fault. There is possibly to make it deterministic, but it is 3x slower.

duncanriach · 2019-10-08T22:20:58Z

For up-to-date status of work on making TensorFlow operate deterministically on GPUs, please see the following repo: https://github.com/NVIDIA/tensorflow-determinism.

Imported from GitHub PR openxla/xla#10636 Add stream id in the backend_config of copy-start instruction. The stream id is obtained from hlo_query::NextChannelId(). The corresponding copy-done instruction which is the use of copy-start instruction will be traversed and added the stream id in the backend_config too. This part is automatically done by the subsequent AnnotateStreamAttributesForUsers() existing in the function. The bool data member copy_start_done_ is used to differentiate copy-start/copy-done from other collective instructions and go through two different paths. openxla/xla#10450 is split and the current PR is the first 1 out of 3 PRs. Copybara import of the project: -- ff99c161a634b868d4265204d10d0b80adf2e772 by Jane Liu <janeliu@nvidia.com>: Add annotation of stream id for copy-start and its use of copy-done instruction -- 64c746a5de6aa4c9370034ed712192dfd78e3a6e by Jane Liu <janeliu@nvidia.com>: Enable the annotator for copy-start/copy-done in gpu compiler -- 7972b987e45cd165834483c4350e953094b5dbe8 by Jane Liu <janeliu@nvidia.com>: Add the annotator pass after HLO rematerialization pass -- 2a9e317aac85e4de3f0ff5f1558885408ab4562d by Jane Liu <janeliu@nvidia.com>: Add the dependency in BUILD -- db5ed79f0ec90a1032252beff5108e4f01b61c4a by Jane Liu <janeliu@nvidia.com>: Use a function to annotate copy-start and add description -- b4b3932bfedfabc7f3e22a8766ecbc1fa1188402 by Jane Liu <janeliu@nvidia.com>: remove the bool var copy_start from the StreamAttributeAnnotator class -- 11148fe63588f58c85ab2530bda64807bf400476 by Jane Liu <janeliu@nvidia.com>: Fixes according to the code review Merging this change closes #10636 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#10636 from zhenying-liu:offloading/annotator 11148fe63588f58c85ab2530bda64807bf400476 PiperOrigin-RevId: 617425257

Imported from GitHub PR openxla/xla#10636 Add stream id in the backend_config of copy-start instruction. The stream id is obtained from hlo_query::NextChannelId(). The corresponding copy-done instruction which is the use of copy-start instruction will be traversed and added the stream id in the backend_config too. This part is automatically done by the subsequent AnnotateStreamAttributesForUsers() existing in the function. The bool data member copy_start_done_ is used to differentiate copy-start/copy-done from other collective instructions and go through two different paths. openxla/xla#10450 is split and the current PR is the first 1 out of 3 PRs. Copybara import of the project: -- ff99c161a634b868d4265204d10d0b80adf2e772 by Jane Liu <janeliu@nvidia.com>: Add annotation of stream id for copy-start and its use of copy-done instruction -- 64c746a5de6aa4c9370034ed712192dfd78e3a6e by Jane Liu <janeliu@nvidia.com>: Enable the annotator for copy-start/copy-done in gpu compiler -- 7972b987e45cd165834483c4350e953094b5dbe8 by Jane Liu <janeliu@nvidia.com>: Add the annotator pass after HLO rematerialization pass -- 2a9e317aac85e4de3f0ff5f1558885408ab4562d by Jane Liu <janeliu@nvidia.com>: Add the dependency in BUILD -- db5ed79f0ec90a1032252beff5108e4f01b61c4a by Jane Liu <janeliu@nvidia.com>: Use a function to annotate copy-start and add description -- b4b3932bfedfabc7f3e22a8766ecbc1fa1188402 by Jane Liu <janeliu@nvidia.com>: remove the bool var copy_start from the StreamAttributeAnnotator class -- 11148fe63588f58c85ab2530bda64807bf400476 by Jane Liu <janeliu@nvidia.com>: Fixes according to the code review Merging this change closes #10636 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#10636 from zhenying-liu:offloading/annotator 11148fe63588f58c85ab2530bda64807bf400476 PiperOrigin-RevId: 617496055

Imported from GitHub PR openxla/xla#10636 Add stream id in the backend_config of copy-start instruction. The stream id is obtained from hlo_query::NextChannelId(). The corresponding copy-done instruction which is the use of copy-start instruction will be traversed and added the stream id in the backend_config too. This part is automatically done by the subsequent AnnotateStreamAttributesForUsers() existing in the function. The bool data member copy_start_done_ is used to differentiate copy-start/copy-done from other collective instructions and go through two different paths. openxla/xla#10450 is split and the current PR is the first 1 out of 3 PRs. Copybara import of the project: -- ff99c161a634b868d4265204d10d0b80adf2e772 by Jane Liu <janeliu@nvidia.com>: Add annotation of stream id for copy-start and its use of copy-done instruction -- 64c746a5de6aa4c9370034ed712192dfd78e3a6e by Jane Liu <janeliu@nvidia.com>: Enable the annotator for copy-start/copy-done in gpu compiler -- 7972b987e45cd165834483c4350e953094b5dbe8 by Jane Liu <janeliu@nvidia.com>: Add the annotator pass after HLO rematerialization pass -- 2a9e317aac85e4de3f0ff5f1558885408ab4562d by Jane Liu <janeliu@nvidia.com>: Add the dependency in BUILD -- db5ed79f0ec90a1032252beff5108e4f01b61c4a by Jane Liu <janeliu@nvidia.com>: Use a function to annotate copy-start and add description -- b4b3932bfedfabc7f3e22a8766ecbc1fa1188402 by Jane Liu <janeliu@nvidia.com>: remove the bool var copy_start from the StreamAttributeAnnotator class -- 11148fe63588f58c85ab2530bda64807bf400476 by Jane Liu <janeliu@nvidia.com>: Fixes according to the code review Merging this change closes #10636 PiperOrigin-RevId: 617550093

Non-determinism Docs (tensorflow#2732)

81ebece

Wrote a short tutorial on non-determinism in TensorFlow due to GPU reductions.

googlebot added the cla: yes label Jun 11, 2017

martinwicke assigned michaelisard Jun 13, 2017

martinwicke added the awaiting review Pull request awaiting review label Jun 13, 2017

michaelisard assigned zheng-xq and unassigned michaelisard Jun 14, 2017

zheng-xq assigned ekelsen and unassigned zheng-xq Jun 15, 2017

drpngx closed this Jun 26, 2017

alexklibisz mentioned this pull request Jul 17, 2017

UNet2DS tensorflow non-deterministic training alexklibisz/deep-calcium#1

Open

drpngx mentioned this pull request Sep 8, 2017

About Deterministic Behaviour of GPU implementation of tensorflow #12871

Closed

jart mentioned this pull request Mar 31, 2018

Feature Request: Support for configuring deterministic options of cudNN Conv routines #18096

Closed

jkschin mentioned this pull request Apr 25, 2018

Mention that GPU reductions are nondeterministic in docs #2732

Closed

This was referenced Aug 14, 2018

BUG: optimizer.compute_gradients() produces inconsistent gradient with the same training instance and label #21515

Closed

BUG: Initial values are still random, even both the graph-level and the operation seed are set #21432

Closed

copybara-service Bot mentioned this pull request Mar 20, 2024

PR #10636: Offloading 1/3: Add annotation for copy-start/copy-done #64026

Closed

copybara-service Bot mentioned this pull request Mar 20, 2024

PR #10636: Offloading 1/3: Add annotation for copy-start/copy-done #64045

Merged

Navigation Menu

Repository navigation

Non-determinism Docs (#2732) - #10636

Non-determinism Docs (#2732)#10636
jkschin wants to merge 1 commit into
tensorflow:masterfrom
jkschin:non_determinism

jkschin commented Jun 11, 2017 •

edited by martinwicke

Loading

Uh oh!

tensorflow-jenkins commented Jun 11, 2017

Uh oh!

michaelisard commented Jun 14, 2017

Uh oh!

zheng-xq commented Jun 15, 2017

Uh oh!

jkschin commented Jun 21, 2017

Uh oh!

ekelsen commented Jun 21, 2017

Uh oh!

ekelsen commented Jun 21, 2017

Uh oh!

jkschin commented Jun 22, 2017

Uh oh!

drpngx commented Jun 26, 2017

Uh oh!

jkschin commented Jun 27, 2017

Uh oh!

alexklibisz commented Jul 14, 2017

Uh oh!

ekelsen commented Sep 13, 2017

Uh oh!

ekelsen commented Sep 13, 2017

Uh oh!

duncanriach commented Oct 8, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Conversation

jkschin commented Jun 11, 2017 • edited by martinwicke Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tensorflow-jenkins commented Jun 11, 2017

Uh oh!

michaelisard commented Jun 14, 2017

Uh oh!

zheng-xq commented Jun 15, 2017

Uh oh!

jkschin commented Jun 21, 2017

Uh oh!

ekelsen commented Jun 21, 2017

Uh oh!

ekelsen commented Jun 21, 2017

Uh oh!

jkschin commented Jun 22, 2017

Uh oh!

drpngx commented Jun 26, 2017

Uh oh!

jkschin commented Jun 27, 2017

Uh oh!

alexklibisz commented Jul 14, 2017

Uh oh!

ekelsen commented Sep 13, 2017

Uh oh!

ekelsen commented Sep 13, 2017

Uh oh!

duncanriach commented Oct 8, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

jkschin commented Jun 11, 2017 •

edited by martinwicke

Loading