Boris Chow • over 7 years ago
Setup schedule job for Sparkathon
Hi all, how can I retrieve & insert weather data periodically to the db / storage space ? I have already written a python script to retrieve weather data from IBM Weather Inside Service but I am not sure how to set up the scheduler job at Spark service.
Comments are closed.
Serena Pietruszka Manager • over 7 years ago
Thanks for your question.
Unfortunately, there is currently no way to schedule jobs. Notebooks are not designed to work that way as they terminate the session once the page is closed or after a period of inactivity. This feature is one that's in the works (via Spark-Submit) and is set to be available soon, but it may not be in time for the Sparkathon competition.
Hope this helps!