Building ML models in watsonx.ai and batch scoring with Spark in watsonx.data accelerator

The Building ML models in watsonx.ai and batch scoring with Spark in watsonx.data accelerator guides you through the end-to-end workflow of training and persisting ML models in watsonx.ai and batch scoring those models with Spark at scale in watsonx.data.

Try this accelerator to set up an end-to-end workflow of training and persisting ML models in watsonx.ai and then batch score them at scale with Spark in watsonx.data.

To start, log into the Resource hub and then create a new project that is based on the Building ML models in watsonx.ai and batch scoring with Spark in watsonx.data sample project.

This accelerator provides a scalable ML pattern and implementation that uses a pipeline to automate the end-to-end AI lifecycle, from loading data, training models, selecting the best result to persisting the best model in a model repository. Create and configure the flow once and then run it on demand or on a schedule, without having to interact with individual tools and assets.

The following processes are included:

• Loading training data
• Training a scikit-learn ML model or an AutoAI model on a common training data set
• Comparing the results and selecting the best model. Then persisting the model in model repository
• Transforming the model to a Spark model by using UDF and then performing batch scoring at scale on this model

This graph shows a pipeline view of this workflow:

This graph shows a detailed architectural diagram of the scalable ML pattern that is used by the Building ML models in watsonx.ai and batch scoring with Spark in watsonx.data accelerator: