Elena' s AI Blog

TensorFlow: Global and Operation-level Seeds

15 Jan 2022 / 11 minutes to read

Elena Daehnhardt


Midjourney AI-generated art


Introduction

In training Machine Learning models, we want to avoid any ordering biases in the data. In some cases, such as in Cross-Validation experiments, it is essential to mix data and ensure that the order of data is the same between different runs or system restarts. We can use operation-level and global seeds to achieve the reproducibility of results.

Global and Operation-level Seeds

To begin, let’s create a mutable tensor with “Variable.”

# Create a variable tensor
tensor = tf.Variable([[[1, 2, 3],
                       [4, 5, 6]],
                      [[7, 8, 9],
                       [10, 11, 12]],
                      [[13, 14, 15],
                       [16, 17, 18]]])

In the code below, we use assign method to change the first element (which is a matrice) in tensor. We fillled its values with zeros.

# Change elements of the first tensor element
tensor[0].assign([[0, 0, 0], [0, 0, 0]])
<tf.Variable 'UnreadVariable' shape=(3, 2, 3) dtype=int32, numpy=
array([[[ 0,  0,  0],
        [ 0,  0,  0]],

       [[ 7,  8,  9],
        [10, 11, 12]],

       [[13, 14, 15],
        [16, 17, 18]]], dtype=int32)>

In TensorFlow, we have global and operation-level seeds.

The global seed we define with set_seed:

# Set a random seed with value of 57
tf.random.set_seed(57)

The operation-level we can define directly in operation such as when shuffling the tensor

tf.random.shuffle(tensor, seed=57)

I will further go into four scenarios when using two types of seeds in TensorFlow.

No seeds defined

When we do not define any seeds, the system chooses any random seed in its pool. This will result in different values every time we re-run the code. This is why this scenario does not guarantee any reproducibility of results.

tf.random.shuffle(tensor)
<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=
array([[[ 7,  8,  9],
        [10, 11, 12]],

       [[ 0,  0,  0],
        [ 0,  0,  0]],

       [[13, 14, 15],
        [16, 17, 18]]], dtype=int32)>

The second run changes the data order:

tf.random.shuffle(tensor)
<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=
array([[[ 7,  8,  9],
        [10, 11, 12]],

       [[13, 14, 15],
        [16, 17, 18]],

       [[ 0,  0,  0],
        [ 0,  0,  0]]], dtype=int32) >

The data order will differ from the first run when we restart the environment.

tf.random.shuffle(tensor)
<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=
array([[[ 0,  0,  0],
        [ 0,  0,  0]],

       [[ 7,  8,  9],
        [10, 11, 12]],

       [[13, 14, 15],
        [16, 17, 18]]], dtype=int32) >
tf.random.shuffle(tensor)
<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=
array([[[ 0,  0,  0],
        [ 0,  0,  0]],

       [[13, 14, 15],
        [16, 17, 18]],

       [[ 7,  8,  9],
        [10, 11, 12]]], dtype=int32) >

Only global seed defined

We can just define the global seed with the set_seed function. In this case, the system combines the defined global seed and an arbitrary random seed. After the restart, we got the same shuffled data order when we re-ran the code.

tf.random.set_seed(seed=57)
tf.random.shuffle(tensor)
<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=
array([[[ 0,  0,  0],
        [ 0,  0,  0]],

       [[13, 14, 15],
        [16, 17, 18]],

       [[ 7,  8,  9],
        [10, 11, 12]]], dtype=int32) >
tf.random.shuffle(tensor)
<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=
array([[[ 7,  8,  9],
        [10, 11, 12]],

       [[ 0,  0,  0],
        [ 0,  0,  0]],

       [[13, 14, 15],
        [16, 17, 18]]], dtype=int32) >

Thus, we see precisely the same order of data between restarts.

Operation seed only

When defining only operation seed value, TensorFlow uses a default global seed plus the defined operation seed. When rerunning the code, we get the same results. When we restart, we get the same sequence of results.

tf.random.shuffle(tensor, seed=24)
<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=
array([[[ 7,  8,  9],
        [10, 11, 12]],

       [[ 0,  0,  0],
        [ 0,  0,  0]],

       [[13, 14, 15],
        [16, 17, 18]]], dtype=int32) >
tf.random.shuffle(tensor, seed=24)
<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=
array([[[ 0,  0,  0],
        [ 0,  0,  0]],

       [[13, 14, 15],
        [16, 17, 18]],

       [[ 7,  8,  9],
        [10, 11, 12]]], dtype=int32) >

Both seeds are defined

This is the best case to achieve reproducible randomness. We define global and operation-level seeds. We get the same sequence of results also when we restart the environment. Thus, after running the code, we always get the same sequence of results.

tf.random.set_seed(seed=57)
tf.random.shuffle(tensor, seed=75)
<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=
array([[[13, 14, 15],
        [16, 17, 18]],

       [[ 7,  8,  9],
        [10, 11, 12]],

       [[ 0,  0,  0],
        [ 0,  0,  0]]], dtype=int32)>
tf.random.shuffle(tensor, seed=75)
<tf.Tensor: shape=(3, 2, 3), dtype=int32, numpy=
array([[[ 7,  8,  9],
        [10, 11, 12]],

       [[ 0,  0,  0],
        [ 0,  0,  0]],

       [[13, 14, 15],
        [16, 17, 18]]], dtype=int32)>

Did you like this post? Please let me know if you have any comments or suggestions.

Python posts that might be interesting for you



Conclusion

In this post, I have described how we can ensure the required level of reproducibility when using global and operation-level seeds. For writing this post, I have used TensorFlow documentation and tutorials at Udemy, TensorFlow Developer Certificate in 2022: Zero to Mastery.

desktop bg dark

About Elena

Elena, a PhD in Computer Science, simplifies AI concepts and helps you use machine learning.

Citation
Elena Daehnhardt. (2022) 'TensorFlow: Global and Operation-level Seeds', daehnhardt.com, 15 January 2022. Available at: https://daehnhardt.com/blog/2022/01/15/tf-seeds/
All Posts