Yahoo is open sourcing TensorFlowOnSpark. The announcement came today, and the code has already been made available under an Apache 2.0 license on GitHub. Also included in the directory are assorted examples and other relevant resources.
Announcing the move through a blog post, Yahoo said:
Today, we are pleased to offer TensorFlowOnSpark to the community, our latest open source framework for distributed deep learning on big-data clusters.
In case you are unaware of it, TensorFlowOnSpark is Yahoo’s take on Google’s TensorFlow open-source framework. It makes Yahoo’s data sets compatible with the library. What library? Well, TensorFlow is basically an open source software library that allows for numerical computation using data flow graphs. The term “tensor” is significant here because while nodes in the graph equate to mathematical operations, graph edges represent the multidimensional data arrays or tensors that are communicated between the respective nodes.
This is not the first time that Yahoo is open sourcing one of its softwares. Last year, the company open sourced CaffeOnSpark, another of its open source frameworks. CaffeOnSpark actually allowed distributed deep learning and big-data processes on both Spark and Hadoop clusters. That particular software was used to improve NSFW image detection, automatically identify eSports game highlights and so on. So yeah, the software was pretty useful and it was actually quite generous of the company to open source it.
Yahoo’s latest move with regards to TensorFlow appears to have been taken in the same vein.
With the community’s valuable feedback and contributions, CaffeOnSpark has been upgraded with LSTM support, a new data layer, training and test interleaving, a Python API, and deployment on docker containers. This has been great for our Caffe users, but what about those who use the deep learning framework TensorFlow? We’re taking a page from our own playbook and doing for TensorFlow for what we did for Caffe.
The move would do the developer community good considering that TensorFlow and TensorFlowOnSpark programs can easily be changed to work within each other’s frameworks. As per Yahoo, changing fewer than 10 lines of Python code are needed. Yahoo has also provided the community with a variety of examples so as to hit the ground running.
You can read more regarding the announcement and TensorFlow, right here.
